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ABSTRACT 

The  Earth  Sciences  Decadal  Survey  of  2007  presented  a  comprehensive  vision  for 
the  evolution  of  space-based  Earth  Science  resources.  The  practical  development  of  the 
Decadal  campaign,  however,  has  highlighted  four  challenges  to  the  original  plan:  the 
growth  of  expected  costs  and  the  reduction  of  program  budget,  the  loss  and  changing  status 
of  the  expected  precursor  missions,  the  opportunity  afforded  by  international  earth  science 
efforts,  and  the  increasing  desire  to  operationalize  key  measurements  of  the  earth.  This 
thesis  discusses  how  system  architecting  of  the  Decadal  campaign  can  realistically 
reproduce  the  decision  logic  of  the  Decadal  Survey,  while  accurately  capturing  the 
necessary  constraints  and  value  functions,  and  can  form  the  basis  for  rational  analysis  of 
the  effects  of  changing  assumptions. 

This  thesis  presents  a  technique  for  tracing  stakeholder  value  to  campaign 
architecture  decisions  through  a  system  of  science  traceability  matrices.  Using  a 
framework  based  upon  decomposition  of  value-related  elements,  the  costs  and  benefits  of 
the  Decadal  campaign  are  analyzed. 

This  thesis  refines  a  technique  for  the  scheduling  of  space-based  observation 
campaigns  and  provides  insight  and  recommendations  for  the  Earth  Observation  Program. 
The  decision  logic  of  the  Decadal  Survey  is  implemented  through  constraints  and  value 
functions,  and  an  algorithm  for  scheduling  is  developed.  Finally,  this  algorithm  is  used  to 
examine  the  impacts  of  key  changes  that  have  occurred  since  the  publishing  of  the  Decadal 
Survey  and  provide  recommendations  for  the  development  of  the  Earth  Science  Decadal 
Survey  campaign. 
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“The  committee  took  as  its  charge  the  provision  of  a  strategy  for  a  strong,  balanced 
national  program  in  Earth  science  for  the  next  decade  that  could  be  carried  out  with 
what  are  thought  to  be  realistic  resources.  Difficult  choices  were  inevitable,  but  the 
recommendations  presented  in  this  report  reflect  the  committee’s  best  judgment, 
informed  by  the  work  of  the  panels  and  discussions  with  the  scientific  community,  about 
which  programs  are  most  important  for  developing  and  sustaining  the  Earth  science 
enterprise.  ’’ 

— National  Research  Council  Report,  Earth  Science  and  Applications  from  Space: 
National  Imperatives  for  the  Next  Decade  and  Beyond 


1  Introduction 

1.1  Motivation 

On  12  January  2006,  The  Secretary  of  the  Air  Force  formally  notified  Congress  that 
the  National  Polar-orbiting  Operational  Environmental  Science  System  (NPOESS)  program 
had  breached  Nunn-McCurdy  certification  by  cost-overruns  of  greater  than  25%.  This 
allowed  for  three  options:  cancelling  the  program,  certifying  a  restructured  program,  or 
certifying  the  program  with  updated  cost  and  schedule.  Faced  with  rapidly  increasing 
costs,  the  Integrated  Program  Office,  a  joint  venture  between  NOAA  and  the  US  Air  Force, 
and  the  entity  responsible  for  the  NPOESS  program,  chose  to  restructure — saving  the 
program,  but  significantly  reducing  its  capabilities. 

Ironically,  the  IPO  was  originally  commissioned  as  a  cost-saving  effort  to  reduce 
redundant  engineering  programs.  Its’  purpose  was  simple:  to  design  and  build  a  polar- 
orbiting  weather  satellite  network  that  could  be  used  by  a  multitude  of  federal  agencies, 
including  the  DoD,  NOAA,  NASA,  USGS,  and  others.  However,  as  costs  rose,  the  NPOESS 
recertification  required  a  reduction  of  capability,  or  elimination,  of  various  instrument 
packages  from  the  program.  Of  the  38  Environmental  Data  Records  the  mission  was 
originally  designed  to  capture,  21  were  either  demanifested  or  significantly  degraded.  The 
silent  casualty  in  the  process  was  NASA’s  Earth  Science  program,  which  had  been  counting 
on  getting  key  sets  of  data  from  the  mission. 
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In  2004  the  National  Research  Council  commissioned  a  Decadal  Survey  report  for 
earth  science  to  establish  a  first  unified  agenda  for  space-based  observations.  Much  like 
the  original  vision  of  NPOESS,  a  long-term  agenda  would  promote  cost-sharing 
opportunities  and  eliminate  redundancies  while  cementing  the  societal  benefits  of  earth 
science.  Not  only  could  more  be  done  with  less,  but  in  demonstrating  its  applicability  to 
society,  the  community  could  ensure  the  continuing,  and  expanding,  interest  of  its 
stakeholders.  Such  a  plan  would  be  a  tremendous  return  on  investment — a  few  months  of 
discussion  in  return  for  decades  of  program  support. 

The  report  was  released  three  years  later,  and  in  many  ways  lived  up  to  its  potential. 
It  carefully  laid  out  a  campaign  of  17  earth  science  space  missions  that,  while  primarily  of 
an  experimental  nature,  could  all  have  strong  practical  applications,  and  carry  the 
possibility  of  being  continued  as  long-term  operational  programs.  These  missions  spanned 
the  spectrum  of  earth  sciences,  and  represented  the  goals  and  objectives  of  many  different 
sub-communities.  The  report  marked  a  monumental  consensus-building  victory  for  the 
earth  science  community,  which  had  never  outlined  a  unified  picture  of  their  priorities. 
Perhaps  most  importantly,  the  report  gave  NASA  and  NOAA  an  endorsed  plan  for  the  next 
decade. 

While  the  Decadal  Survey  marked  an  important  step  for  the  earth  science 
community,  its  current  utility  is  waning.  A  few  major  assumptions,  such  as  the  status  of 
expected  precursor  missions  (including  NPOESS),  the  costing  of  the  17  proposed  missions, 
and  the  budget  allocated  to  earth  science,  have  changed.  By  current  estimates  the 
"Decadal"  Survey  will  take  more  than  40  years  to  complete. 

The  priorities  established  by  the  Decadal  Survey  are  impossible  to  ignore,  yet 
commissioning  a  new  study  every  time  assumptions  change  is  impractical.  A  formal 
system  for  re-architecting  the  Decadal  campaign  is  necessary,  one  that  is  practical  and 
responsive  to  the  changes  that  have  occurred.  The  Decadal  Survey  is  the  basis  for  this 
thesis,  which  attempts  to  maintain  the  priorities  and  objectives  of  the  Survey  while  re¬ 
structuring  the  contents  of  the  17  Missions  according  to  updated  constraints  and 
assumptions. 
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1.2  The  Decadal  Survey 


The  Decadal  Survey  was  commissioned  with  five  specific  tasks  in  its  pursuance  of  a  unified 
agenda  for  earth  science  (Figure  1).  These  tasks  loosely  fall  into  three  questions:  "Where 
are  we?",  "Where  are  we  going?",  and  "How  will  we  get  there?" 


1 .  Review  the  status  ot  the  field  to  assess  recent  progress  in  resolving  major  scientific  questions  outlined 
in  relevant  prior  NRC,  NASA,  and  other  relevant  studies  and  in  realizing  desired  predictive  and  applica¬ 
tions  capabilities  via  space-based  Earth  ohserwations; 

2.  Develop  a  consensus  of  the  top-level  scientific  questions  that  should  provide  the  focus  for  Earth 
and  environmental  observations  in  the  period  2005-2015; 

3.  Take  into  account  the  principal  federal-  and  state-level  users  of  these  obser\'ations  and  identify  oppor¬ 
tunities  for  and  challenges  to  the  exploitation  of  the  data  generated  by  Earth  obsers'ations  from  space; 

4.  Recommend  a  prioritized  list  of  measurements,  and  identify  potential  new  space-based  capabilities 
and  supporting  activities  within  NASA  ESE  |Earth  Science  Enterprise]  and  NOAA  NESDIS  to  support  national 
needs  for  research  and  monitoring  of  the  dynamic  Earth  system  during  the  decade  2005-201 5;  and 

5.  Identify  important  directions  that  should  influence  planning  for  the  decade  beyond  201 5. 

Figure  1.  Decadal  Survey  Tasks  (National  Research  Council,  2007) 

First,  the  Decadal  Committee  wanted  to  understand  where  the  earth  science 
community  stood.  To  this  end,  dozens  of  leading  scientists  and  engineers  were  invited  to 
participate.  Divining  the  status  of  communities  was  a  relatively  straightforward  matter, 
since  current  resources  and  applications  are  easily  enumerable.  Simply  inviting  a  few 
participants  from  key  federal  agencies  would  have  been  sufficient  to  accurately  capture  the 
current  status  of  earth  science  from  space. 

Second,  the  Decadal  Committee  wanted  to  understand  where  the  earth  science 
community  should  be  going.  This  was,  perhaps,  the  most  controversial  aspect  of  the 
survey,  as  synthesizing  a  vision  required  uniting  disparate  perspectives.  Seven  thematic 
panels  were  commissioned  to  represent  different,  but  not  homogeneous,  interests: 

•  Earth  science  applications  and  societal  benefits 

•  Human  Health  and  security 

•  Land-use,  ecosystems,  and  bio-diversity 

•  Solid  Earth  Hazards,  natural  resources,  and  dynamics 

•  Climate  variability  and  change 

•  Weather  Science  and  Applications 

•  Water  resources  and  the  global  hydrological  cycle 
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The  six  science  panels  were  individually  given  the  tasks  of  coming  up  with  a  set  of 
recommended  space  missions.  While  the  Decadal  Committee  had  send  out  a  request  for 
proposals  to  the  larger  scientific  community  and  had  a  list  of  specifically  proposed 
missions,  first  each  panel  had  to  agree  upon  its  priorities.  Although  each  panel  conducted 
this  exercise  independently,  the  general  process  involved  identifying  science  themes  and 
the  key  questions  relevant  to  those  themes  (Figure  2). 


BOX  7.2  SCIENCE  THEMES  AND  KEY  QUESTIONS  FOR  IDENTIFYING  PRIORITIES  FOR  SATELLITE 
OBSERVATIONS  FOR  UNDERSTANDING  AND  MANAGING  ECOSYSTEMS 


Science  Themes  Key  Questions 

How  does  climate  change  affect  the  carbon  cycle? 

How  does  changing  terrestrial  water  balance  affect  carbon  storage  by  terrestrial 
ecosystems? 

How  do  increasing  nitrogen  deposition  and  precipitation  affect  terrestrial  and  coastal 
ecosystem  structure  and  function  and  contribute  to  climate  feedbacks? 

How  do  large-scale  changes  in  ocean  circulation  affect  nutrient  supply  and 
ecosystem  structure  in  coastal  and  off-shore  ecosystems? 

How  do  Increasing  inputs  of  pollutants  to  freshwater  systems  change  ecosystem 
function? 

What  are  the  management  opportunities  for  minimizing  disruption  in  carbon, 
nitrogen,  and  water  cycles? 

Figure  2.  Decadal  Survey  Process  (National  Research  Council,  2007) 

Finally,  at  this  point  in  the  Decadal  process,  the  questions  "Where  are  we  going?" 
and  "How  will  we  get  there?"  began  to  merge.  Each  key  question  identified  by  a  panel  was 
linked  with  an  answer — a  set  of  measurements  that  could  possibly  answer  that  question, 
and  a  set  of  instruments  that  could  capture  those  measurements.  Questions  with  similar 
answers  were  combined  together  to  form  specific  objectives,  objectives  that  could 
subsequently  be  prioritized.  Each  panel  then  reviewed  the  submitted  mission  proposals 
for  those  that  satisfied  their  objectives,  and  came  up  with  their  own  proposed  campaign. 
The  Decadal  Committee  then  took  the  proposed  campaigns  of  every  panel  and  combined 
them.  Different  mission  proposals  were  tied  together,  others  were  left  out.  Compromises 
were  reached  that  reduced  the  35  proposed  panel  missions  into  one  campaign  of  17 
missions. 


Disruption  of  the 
Carbon,  Water,  and 
Nitrogen  Cycles 
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The  process  of  reaching  this  answer  included  the  work  of  over  a  hundred  people 
over  the  course  of  three  years.  Although  of  monumental  utility  to  the  earth  science 
community,  tracing  the  decisions  that  led  to  the  final  campaign  is  an  impossible  process. 
Too  many  tradeoffs  and  compromises  occurred  before  making  it  into  the  final  report.  This 
complicates  and  confuses  the  campaign  architect  now  trying  to  implement  this  agenda — he 
has  little  idea  what  all  the  decisions  were,  and  even  less  idea  what  assumptions  actually 
influenced  those  particular  decisions. 

1.3  General  Objective 

The  general  objective  of  this  thesis  is  as  follows: 

•  Capture  the  Decadal  Survey  decisions  processes  and  logic  for  automated  and 
optimizable  architecture  development  under  changing  assumptions. 

1.4  Framework  for  Analysis 

The  process  of  system  architecting  requires  a  holistic  consideration  of  "the  system". 
While  some  systems,  particularly  simple  ones,  can  assume  isolation  during  the  architecting 
process,  to  do  so  with  complex,  large-scale  projects  is  to  invite  disaster.  Developing  a 
comprehensive  framework  to  describe  the  technical,  social,  economic,  and  political 
environments  and  limitations  that  encompass  a  project  is  a  necessary  architecting  step. 

Tim  Sutherland  of  the  MIT  Space  Architecture  group  described  a  framework  for  analyzing 
the  Earth  Observation  program  (Figure  3).  In  this  framework  tacit  relationships  are 
defined  in  context  to  the  production  of  value,  and  different  levels  of  abstraction  are  used  to 
describe  intermediate  steps.  For  this  thesis  value  is  defined  as  benefit  produced  at  some 
cost. 
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Value  Delivery 

Stakeholders 

Stakeholder 

Needs 

Sciences  Policy 

Objectives  Measurements 

Operations 
Instruments  &  Support 

Figure  3.  Framework  for  analysis  of  stakeholders  and  system  architecture  for  the 
Earth  Observations  Program  (Sutherland,  2009) 

At  its  core  this  framework  depicts  value  delivery  on  a  spectrum.  On  the  left  side  of 
the  figure  benefit  is  realized,  while  on  the  right  costs  are  incurred.  The  farthest  left  column, 
Stakeholders,  represent  the  distinct  stakeholder  and  beneficiary  groupings  that  Sutherland 
identified  as  central  to  the  Earth  Observation  Program.  While  the  highest  level  of 
abstraction,  each  of  these  Stakeholders  has  a  tacit  list  of  needs.  These  needs  are  more 
directly  expressed  by  Stakeholders  through  Objectives.  For  the  case  of  Earth  Scientists  in 
Figure  3  these  Objectives  are  satisfied  via  the  data  produced  through  measurements. 
Measurements  generated  through  the  physical  operations  of  instruments,  and  instruments 
are  supported  by  the  bevy  operational  requirements  of  a  space  program.  Using  this 
framework,  the  complexity  of  the  Earth  Observation  system  can  be  decomposed  and 
understood  on  a  component  level. 
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Justin  Colson,  also  of  the  MIT  Space  Architecture  group,  wrote  his  Master’s  thesis 
about  the  application  of  this  framework  to  rescheduling  the  Decadal  Survey  missions.  He 
demonstrated  a  modeling  technique  for  satellite  network  scheduling  that  considered 
flexibility  on  in  the  rightmost  column  of  this  framework,  varying  only  launch  date.  This 
thesis  builds  his  work  and  expands  the  architecting  process  to  the  intermediate  columns  of 
this  framework. 

1.5  Justification  for  Rearchitecting  the  Decadal  Survey  Campaign 

Although  the  changes  to  NPOESS  have  affected  the  preservation  of  data  records  in 
an  unforeseen  manner,  the  motivation  for  rearchitecting  the  Decadal  campaign  is 
multifaceted.  Four  factors  reflect  the  change  of  assumptions  since  the  Decadal  Survey:  cost 
growth  and  budget  limitations,  the  increasing  need  for  operational  programs,  the  status  of 
expected  precursors  missions,  and  the  contributions  of  international  agencies. 

The  Decadal  campaign  relies  primarily  on  NASA  for  implementation  (fourteen  and  a 
half  of  the  seventeen).  Recently,  the  Congressional  Budget  Office  published  a  report 
highlighting  projected  budget  growth  (Figure  4). 
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Figure  4.  NASA  Earth  Science  Budget  Forecast  (CBO,  2009) 
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Only  about  a  quarter  of  the  earth  science  budget,  or  about  $300M/year,  is  projected  to  be 
directed  towards  the  Decadal  Survey  campaign  (NASA,  2008).  One  of  the  key  assumptions 
of  the  Decadal  Survey  was  that  the  budget  would  return  to  FYOO  levels — approximately 
$750M/year.  Assuming  a  best  case  linear  translation  of  time  and  cost,  this  extends  the 
"decadal"  campaign  by  more  than  a  factor  of  two.  Although  the  current  administration 
appears  to  favor  the  Decadal  program,  as  signified  by  the  $400M  boost  from  the  stimulus 
act  specifically  for  Decadal  Survey  mission  development,  this  has  yet  to  impact  long-term 
budgetary  decisions  (Public  Law  111  -  5  -  American  Recovery  and  Reinvestment  Act  of 
2009,2009). 

The  budget  issue  is  further  compounded  by  the  propensity  for  NASA  missions  to 
grow  significantly  in  cost.  A  recent  report  by  the  Government  Accountability  Office 
highlighted  this  trend  (Figure  5). 


Cost  Grow  til  for  72  of  NASA's  Programs 
(Percentage  of  cost  grov/th) 


Figure  5.  Historical  Cost  Growth  of  NASA  Missions  (GAO,  2009) 

The  issue  is  further  complicated  by  the  accounting  used  in  cost  estimation — these  trends 
are  only  reflective  of  cost  after  the  baseline  has  been  established — and  do  not  reflect  the 
cost  growth  in  pre  phase-A  development.  The  Decadal  Survey  estimated  a  campaign  cost  of 
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totaling  around  $7.5B;  NASA’s  initial  cost  estimate  increased  that  number  by  49%  (Volz, 
2008). 

Further  prompting  an  analysis  of  the  Decadal  Survey  is  the  increased  need  to 
integrate  the  Decadal  Survey  program  into  a  long  term  earth  observation  campaign.  The 
success  of  the  current  earth  observation  program  has  exposed  the  scientific  need  for  multi- 
decadal  records.  Additionally,  earth  science  is  starting  to  be  recognized  as  an  essential 
component  of  national  security,  and  is  attracting  interest  from  the  DoD  and  CIA  (CNA, 
2007).  While  the  Decadal  Survey  missions  tend  to  be  experimental,  there  is  a  rising 
urgency  to  establish  an  operational  earth  science  program. 

Additionally,  the  status  of  planned  US  missions  is  constantly  changing.  The  loss  of 
the  Orbiting  Carbon  Observatory  (OCO)  due  to  launch  failure  in  2009  puts  into  serious 
question  the  second  tier  ranking  for  the  Decadal  mission  ASCENDS,  which  will  also 
measure  atmospheric  carbon.  The  Decadal  Survey  expected  certain  missions  to  fly  and,  as 
changes  are  made  to  different  programs,  there  must  be  a  way  to  incorporate  these 
decisions  into  future  planning. 

Finally,  one  area  the  Decadal  Survey  directed  campaign  planners  towards  was  the 
inclusion  of  international  efforts.  The  same  observation  needs  that  the  Decadal  planners 
foresaw  are  shared  by  other  agencies  world-wide.  As  international  earth  science  missions 
are  proposed  and  scheduled,  it  is  extremely  desirable  that  campaign  planning  include  these 
missions.  Leveraging  international  efforts  reduces  the  technology  development  and 
budgetary  risk  factors  to  both  NASA  and  NOAA. 
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1.6  Background  of  Relevant  Literature 


1.6.1  Science  Traceability 

While  the  concept  of  managing  different  levels  of  requirements  has  long  been  a 
central  tent  of  systems  engineering,  only  recently  has  a  formalized  system  been  mandated 
for  NASA  science  missions.  The  Science  Traceability  Matrix  (STM),  as  proposed  by  Weiss, 
Smythe,  and  Lu,  in  Science  Traceability,  offers  a  simple  and  logical  method  for  conveying 
how  specific  mission  requirements  flow  down  from  high  level  goals  (Figure  6). 


1  Program  Objectives  1 

1 

1  Mission  and  Science  Objectives  1 

1 

1  Measurement  Objectives  1 

1 

1  Measurement  Requirements  1 

1 

1  Instrument  Requirements  1 

1 

1  Science  Products  1 

\ 

Space  Craft  Requirements 

Ground  System  Requirements 
Mission  Requirements 

Mission  Operations  Requirements 

Figure  /.  The  contents  of  the  science  matrix  include 
all  science  elements  that  affect  resource  trades  for 
mission  implementation  -which  is  not  part  of  the  STM 
is  summarized  in  the  bottom  box  of  this  figure. 

Figure  6.  STM  flow-down  (Weiss,  Smythe,  &  Lu,  2005) 

Starting  with  high-level  objectives,  such  as  those  enumerated  in  National  Academy 
of  Science  decadal  surveys  or  NASA  strategic  roadmaps,  the  explicit  linkages  between 
different  levels  of  requirements  can  be  enumerated.  As  seen  in  Figure  7,  each  successive 
layer  of  the  STM  provides  increasingly  detailed  requirements. 
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Figure  7.  Example  STM  (Weiss,  Smythe,  &  Lu,  2005) 


A  well  put-together  STM,  one  in  which  requirements,  and  even  expected 
publications,  are  clearly  traceable  to  top-level  objectives,  is  indicative  a  mature  mission 
design.  This  is  an  invaluable  tool  during  the  formulation  phase  of  mission  design,  as  it 
essentially  presents,  in  one  table,  a  complete  justification  for  the  mission.  As  a  mission 
moves  through  its  design  cycle,  the  STM  again  proves  useful  by  defining  a  trade  space; 
adjustments  to  mission  capabilities  can  be  traced  to  requirements,  performance  measures, 
cost  evaluations,  and  ultimately  mission  feasibility. 

in  the  earth  science  Decadal  Survey  each  mission  proposal  was  required  to  include  a 
STM.  However,  most  of  these  proposed  missions  did  not  reach  the  final  campaign  intact,  on 
average  being  a  compilation  of  three  different  proposals.  The  compromises  panels  made 
were  not  enumerated — and  new  STM's  for  each  mission  were  not  included  in  the  final 
report.  The  Decadal  Survey  does  not  reflect  the  detailed  traceability  required  of  an  STM. 

STMs  are  not  inherently  well-suited  to  campaign  analysis.  One  of  the  principles  of 
the  STM  is  simplicity — it  should  convey  all  the  requisite  information  on  just  one  chart; 
conveying  the  same  information  for  a  set  of  17  missions  is  beyond  the  scope  of  the  STM 
methodology.  Additionally,  although  the  STM  is  useful  during  the  entire  mission  lifecycle,  it 
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is  a  time-independent  display.  Dealing  with  scheduling  over  time  is  also  beyond  its  scope. 
Finally,  the  STM  is  insufficient  for  campaign  planning  because,  although  it  enables  analysis 
of  the  impacts  of  programmatic  changes,  it  provides  "no  objective  algorithm  to  quantify  the 
relative  merits  of  high  level  goals"  (Weiss,  Smythe,  &  Lu,  2005).  The  effects  of  decisions  can 
be  traced,  but  for  most  missions  the  importance  of  those  decisions  remains  unknown. 

1.6.2  Global  View  of  Earth  Science 

While  there  are  many  models  describing  the  "earth  system"  perhaps  the  most  well- 
known  is  Bretherton’s  model  (Figure  8).  In  this  figure  he  presents  a  conceptual  model  for 
understanding  global  climate  change  as  a  set  of  interrelated  modules.  Processes, 
feedbacks,  and  forcings  are  holistically  linked  together  in  an  attempt  to  understand  the 
impacts  of  human  activities  within  the  context  of  natural  variability.  Bretherton  makes  the 
observation  that  the  earth  system  is  dependent  both  on  the  physical  climate  system  and 
biogeochemical  cycles.  Hence,  issues  relevant  to  one  field  can  be  equally  important  to 
another,  seemingly  unrelated  field.  For  the  Decadal  panels  the  interests  of  one  are  likely 
shared  by  several  others.  This  is  particularly  true  of  the  Human  Health  and  Security  panel, 
which  did  not  recommend  a  dedicated  mission,  instead  endorsing  other  missions  it  could 
benefit  from. 


Figure  8.  Simplified  Earth  System  Model  (REMOTE  SENSING) 
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Bretherton  also  recognizes  the  need  for  space-based  remote  sensing  in 
understanding  the  earth  system.  Space  platforms  allow  for  both  rapid,  wide  coverage  and 
long-duration,  repetitive  observations.  They  are,  however,  most  effective  when  multiple 
observations  of  the  same  phenomena  can  be  captured. 

Elachi  and  Van  Zyl  observed  that  a  vast  majority  of  remote  sensing  falls  into  a 
spectrum  seen  in  Figure  9  (Elachi  &  Van  Zyl,  2006).  Just  as  the  physical  systems  being 
measured  express  a  large  degrees  of  interrelatedness,  the  instrumentation  necessary  to 
observe  those  systems  are  quite  similar.  Subtle  changes  to  instrument  parameters  can 
widely  vary  the  measurements  captured. 


Spatial 

Information 


Spectral  Intensity 

Information  Information 

Figure  9.  Remote  Sensing  Instruments  and  Information  (Elachi  &  Van  Zyl,  2006) 

While  this  facet  of  remote  sensing  does  hint  at  possible  multi-user  approaches  towards 
instrument  development,  it  also  highlights  the  difficulty  in  quantifying  small  instrument 
differences,  it  takes  expert  knowledge  of  these  sensors  to  understand  how  changes  to 
mission  parameter  will  affect  them.  While  remote  sensing  remains  an  extremely  valuable 
tool  for  understanding  the  earth  system,  it  can  be  difficult  to  accurately  capture  the 
qualities  of  sensors  for  system  architecting. 
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1.6.3  Stakeholder  Value  Network  Analysis 

The  framework  for  analysis  presented  in  Figure  3  indicates  that  benefit  is  realized  at 
the  stakeholder  level.  Hence,  considering  value  in  campaign  design  must  include  some 
discussion  of  stakeholders.  Sutherland’s  thesis  provides  insight  into  the  stakeholders  of 
the  Decadal  Survey  campaign  through  a  value  network  analysis. 


A  value  network  presents  a  formal  system  for  understanding  stakeholder  needs  and 
relationships.  First,  individual  stakeholders  are  identified  with  respect  to  the  reference 
enterprise  (Figure  10).  With  the  Decadal  campaign  this  refers  to  the  joint  NASA/NOAA 
efforts. 


Figure  10.  The  Stakeholders  of  a  NASA/NOAA  Earth  Ohservation  Campaign 

(Sutherland,  2009) 


Sutherland  observed  that  the  stakeholders  of  the  Decadal  campaign  have  four  general 
roles.  Stakeholders  such  as  the  federal  government  make  policy  decisions  such  as 
direction  and  funding  levels.  These  decisions  affect  the  federal  agencies  that  perform  the 
role  of  data  acquisition.  The  acquired  data  then  is  conveyed  to  data  users,  which  use  it  to 
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generate  knowledge.  This  knowledge  then  passes  to  various  other  beneficiaries,  who 
interpret  it.  In  turn  these  interpretations  inform  policy  decisions. 

Stakeholders  are  defined  by  not  only  by  their  roles,  but  also  by  their  specific 
objectives  and  needs.  Understanding  these  needs  is  essential — it  is  in  satisfying  them  that 
benefit  is  generated.  Figure  11  presents  the  characterization  of  Scientists  in  the  Decadal 
campaign  network.  At  a  macroscopic  level,  the  role  of  scientists  is  to  generate  knowledge 
from  raw  data.  They  have  a  specific  set  of  objectives  for  doing  this,  which,  in  turn,  reflects  a 
set  of  needs  from  other  stakeholders. 


Inputs: 

•  Space  acquired  data  (NASA,  NOAA,  Int'l  Partners) 

•  Access  to  space  systems  (NASA/NOAA,  Int'l  Partners) 

•  Science  funding  (NASA,  NOAA,  Agencies) 

’  Skilled  workforce  (Educators) 

•  Future  plans  information  (NASA/NOAA) 

•  Informative  &  entertaining  content  (Media) 


Figure  11.  Stakeholder  Definition  Process  (Sutherland,  2009) 


Scientists 


Role:  Use  Earth  observation  data  to  generate  Earth  science  knowledge, 
develop  science  systems,  and  provide  opinions  to  the  Government  and 
Science  Advisory  Bodies 


Objectives: 

'  Produce  useful  knowledge  and  information  for  society 

•  Advocate  for  specific  scientific  capabilities  from  Earth  observing 
satellites 

•  Provide  advice  to  others  on  matters  of  scientific  interest 

•  Achieve  professional  recognition 

^  Specific  Needs: 

•  Space  acquired  data 

•  Other  requisite  complementary  data 

•  Access  to  existing  and  future  space  systems 

•  Funding 

•  Skilled  and  motivated  workforce 

•  Knowledge  of  NASA/NOAA  objectives,  capabilities,  &  future  plans 

^  *  General  knowledge  and  information  (scientific,  technical,  social,  etc.) 


The  value  network  methodology  assumes  that  a  stakeholder  network  is  closed,  and  that 
every  stakeholder  need  is  somehow  fulfilled  by  an  output  of  another  stakeholder. 

Scientists,  for  example,  need  to  acquire  funding  from  somewhere.  The  value  network,  then, 
is  a  physical  mapping  of  the  outputs  of  one  stakeholder  leading  to  the  inputs  of  another 
(Figure  12). 

These  relationships  can  quantitatively  analyzed.  Sutherland  uses  a  combination  of 
questionnaires  and  supporting  documentation  to  value  each  input  to  a  stakeholder.  The 
linkages  between  stakeholders  can  be  mathematically  combined  to  form  "value  chains", 
which  trace  the  outputs  of  the  reference  stakeholder  through  the  network  back  to  its  own 
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inputs.  In  this  way  the  most  important  relationships  and  inputs  between  the  stakeholders 
can  be  calculated. 


Figure  12.  Simplified  Decadal  Campaign  Stakeholder  Value  Network  (Sutherland, 

2009) 

Figure  12  outlines  the  most  important  stakeholders  and  links  for  the  Decadal 
campaign.  The  top  three  stakeholders  are:  the  government,  which  provides  NASA/NOAA 
with  funding  and  direction;  the  public,  which  provides  the  driving  opinion  and  support 
behind  the  government;  and  scientists,  which  provide  knowledge  to  both  the  government 
and  NASA/NOAA.  The  stakeholders  represented  with  dotted  lines  are  moderately 
important,  whereas  those  that  are  grayed  out  are  relatively  unimportant.  The  most 
important  inputs  to  the  Decadal  campaign  are  government  policy  direction  and  funding. 
The  most  important  outputs  are  space  acquired  data  and  research  funding  to  scientists. 
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1.7  Specific  Objectives 

The  specific  objectives  of  this  thesis  are  as  follows: 

•  To  illustrate  a  method  for  tracing  stakeholder  value  to  campaign  architecture 
decisions  by  developing  a  framework  for  campaign  analysis,  applying  that 
framework  to  the  Decadal  Survey,  and  validating  the  results  of  the  technique 
against  the  Decadal  Survey. 

•  To  refine  a  technique  for  scheduling  space-based  earth  observation  campaigns 
by  developing  constraints  and  value  functions  that  an  algorithm  can  utilize  to 
replicate  the  decision  logic  of  the  Decadal  Survey. 

•  To  provide  insight  and  recommendations  for  the  NASA/NOAA  earth  observation 
program  by  examining  the  impacts  of  key  changes  that  have  occurred  since  the 
publishing  of  the  Decadal  Survey. 

Central  Thesis  Question: 

•  Can  systems  architecting  of  the  Decadal  campaign  reproduce  the  decision  logic  of 
the  Decadal  Survey  and  form  the  basis  for  rational  analysis  of  changing 
assumptions? 


1.8  Overview  of  This  Document 

This  thesis  is  organized  into  six  chapters.  The  remaining  chapters  are  organized  as 
follows: 

•  Chapter  2  presents  an  architectural  framework  for  campaign  development, 
the  Campaign  Science  Traceability  Matrix.  It  defines  and  decomposes  the 
value-related  attributes  of  a  campaign  such  that  stakeholders  needs  can  be 
traced  to  architectural  decisions. 

•  Chapter  3  presents  the  methodology  required  to  populate  the  CSTM  for  the 
Decadal  campaign.  It  presents  the  results  of  stakeholder  valuation  on 
missions,  instruments,  measurements,  and  objectives. 

•  Chapter  4  presents  the  constraints  and  value  functions  that  are  applied  to  the 
CSTM.  It  examines  the  different  classes  of  constraints  and  their  application. 
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It  describes  the  development  of  an  algorithm  for  campaign  scheduling  given 
the  CSTM. 

•  Chapter  5  examines  different  permutations  of  the  baseline  Decadal  Survey 
assumptions.  This  includes  using  current  assumptions,  variations  to  the 
annual  budget,  and  the  re-assignment  of  instruments  to  missions. 

•  Chapter  6  presents  the  insights  and  recommendations  drawn  in  the  other 
five  chapters.  It  also  summarizes  areas  for  future  work. 
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2  A  Framework  for  Understanding  Campaign  Value 


This  chapter  traces  the  development  of  a  Campaign-level  Science  Traceability  Matrix 
as  a  framework  for  understanding  campaigns.  A  hierarchical  decomposition  of  the  value- 
related  processes  of  a  campaign  is  presented.  The  elements  of  this  decomposition  are  then 
applied  to  a  Science-Traceability  like  system  of  matrices  to  create  the  CSTM.  Finally,  the 
merits  of  the  CSTM  framework  for  systems  architecting  are  discussed. 

The  objective  of  this  chapter  is: 

Objectives  of  a  framework  for  understanding  campaign  value: 

•  To  illustrate  a  method  for  tracing  stakeholder  value  to  campaign  architecture 
decisions  by  developing  a  framework  for  campaign  analysis 

This  chapter  is  organized  into  the  following  sections: 

•  Section  2.1:  Tracing  value  through  a  campaign.  This  section  a  definition  of 
campaigns  relevant  to  systems  architecting  and  outlines  the  decomposition  of 
the  value-related  processes  of  a  campaign 

•  Section  2.2:  A  Framework  for  campaign  analysis:  the  CSTM.  This  section 
presents  the  CSTM  framework  for  analysis  which  will  be  applied  to  the  Decadal 
campaign  in  Chapter  3. 

2.1  Tracing  Value  through  a  Campaign 

This  section  defines  a  campaign  and  outlines  the  top-level  questions  answered  by  a 
campaign.  It  presents  a  mapping  of  the  value-related  elements  of  a  campaign  described  by 
Sutherland.  It  provides  a  definition  of  value  and  explains  the  processes  undergone  at  every 
level  that  accrue  benefit  and  incur  costs.  These  elements  form  the  basis  for  the  CSTM. 

2.1.1  Defining  a  Campaign 

Creating  a  framework  for  analyzing  the  Decadal  campaign  requires  a  consistent 
definition  of  what  a  campaign  is.  A  survey  of  current  earth-observation  programs  reveals  a 
lack  of  agreement  of  the  common  elements  of  a  campaign.  Sutherland's  definition  of  an 
earth  observation  campaign  is  utilized  to  inform  a  general  discussion  of  campaigns. 
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The  Decadal  Survey  exclusively  refers  to  "campaigns"  as  ground-based  or  airborne 
systematic  research  operations.  NASA's  Earth  Observation  handbook  uses  the  term 
similarly,  but  adds  the  concept  of  a  "validation  campaign",  where  an  instrument  is  pushed 
to  a  higher  TRL  through  a  progression  of  tests  (NASA,  2006).  These  conceptions  imply  a 
progression  over  time  of  individual  instruments,  but  are  never  applied  to  the  entire 
enterprise.  Both  the  Decadal  Survey  and  NASA  instead  describe  their  sets  of  missions  as 
"programs". 

Sutherland  proposed  the  definition  of  an  earth  observation  campaign  located  below. 
At  the  highest  level,  a  campaign  is  composed  of  a  sequence  of  missions.  These  missions 
rely  upon  instruments  to  capture  measurements,  which  in  turn  deliver  value  through 
objectives  to  stakeholders.  This  definition  will  be  utilized  to  describe  campaigns  in  this 
thesis. 

Earth  Observation  campaign: 

•  A  prioritized  sequence  of  Earth-orbiting  missions  containing  instruments  that  produce 
measurements  of  the  Earth,  which  deliver  value  to  a  diverse  range  of  stakeholders  by 
satisfying  specific  scientific  and  societal  objectives. 

This  definition  was  realized  by  first  answering  three  descriptive  questions  (Table  1). 
At  the  highest  level  of  abstraction,  a  complete  campaign  will  address  each  question: 


Campaign  Question 

Campaign  Answer 

What  should  be  done? 

Purpose 

How  should  it  be  done? 

Constraints 

When  should  it  be  done? 

Sequence 

Table  1.  Questions  a  Campaign  Answers 


The  first  question,  "What",  captures  the  purpose  behind  the  campaign  by  relating  the 
campaign  objectives  to  the  priorities,  goals,  and  needs  of  relevant  stakeholders.  The 
second  question,  "How",  captures  the  boundaries  the  campaign  must  fit  in — including  the 
cost,  budget,  TRL,  data  continuity  and  overlap,  and  other  similar  considerations  that  may 
limit  the  design  of  a  campaign.  This  question  can  alternately  be  formulated,  "How  should  it 
not  be  done".  The  final  question  "when",  explains  the  sequence  or  schedule  of  the 
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campaign.  This  can  be  as  simple  as  the  ordering  of  missions,  or  can  be  as  detailed  as 
forcing  overlap  of  specific  measurements. 


Applying  these  three  questions  to  an  Earth  Observation  campaign  reveals  the  two 
primary  components  described  by  Sutherland:  the  missions  and  the  schedule  (Figure  13). 
The  missions  of  a  campaign  represent  a  merging  of  the  purpose  with  constraints,  whereas 
the  schedule  is  the  union  of  the  sequence  with  constraints.  A  viable  campaign  will  hence 
contain  both  missions  and  a  schedule,  which  incorporate  purpose,  constraints,  and 
schedule. 


V/  V/ 


d 

Sc}i0dtilo 

f 

(p 

i  c 

Missions 
'urpose  ar 
onstrainti 

id 

■  (S. 

squence  a 
onstraints 

nd  ] 

J 

J 

Figure  13.  Attributes  of  a  Campaign 


The  duality  of  a  campaign  differentiates  it  from  a  program  or  system.  A  campaign  is 
not  just  a  set  of  missions  flying  at  once — it  is  the  incorporation  of  mission  elements  into  a 
schedule. 


2.1.2  Defining  a  Campaign  Metric 

Every  program  has  a  desirable  end-state.  A  campaign,  due  to  its  time-dependence, 
dictates  an  evolving  desired  state  over  time.  This  desired  state  profile  is  the  metric  against 
which  possible  campaign  designs  are  measured.  For  this  reason  it  is  necessary  to  pick  a 
metric  capable  of  expressing  the  relevant  aspects  of  the  desired-state. 
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The  simplest  measure  of  a  campaign  is  its  value.  The  Lean  Enterprise  defines  value 
as,  "how  various  stakeholders  find  partieular  worth,  utility,  benefit,  or  reward  in  exehange  for 
their  respeetive  eontributions  to  the  enterprise"  (Murman,  2002).  Expressed  simply,  value  is 
benefit  that  is  delivered  at  cost.  It  is  a  useful  metric  in  defining  campaigns  because  of  its 
relation  to  stakeholder  needs — value  exists  only  from  the  perspective  of  the  recipient. 
However,  quantifying  and  comparing  the  value  of  different  things  can  be  difficult, 
particularly  complex  systems  that  service  multiple  stakeholders. 


Definition  of  Value: 

•  Value  is  the  expression  of  benefit  accrued  at  cost  evaluated  from  the  perspective  of 
the  recipient 

Having  decomposed  campaigns  into  missions  and  a  schedule,  the  source  of  value 
can  be  described  at  a  more  specific  level.  Sutherland'  definition  of  a  campaign  can  be 
expressed  as  a  set  of  interrelated  value  elements  (Figure  14).  Each  element  has  a  set  of 
attributes  which  dictate  how  value  is  created  or  modified  by  that  element.  The  remainder 
of  this  section  describes  these  elements  in  detail. 


Figure  14.  Value  Related  Elements  of  a  Campaign 


38 


2.1.3  Schedule 

The  schedule  of  a  campaign  primarily  serves  to  modify  the  value  inherent  in  the 
missions.  Changing  stakeholder  needs,  opportunities  for  data  continuity  and  overlap, 
synergistic  measurements,  and  the  urgency  of  science  objectives  all  affect  scheduling. 
Between  these  characteristics,  different  scientific  communities  will  see  different  levels  of 
value  from  identically  composed,  but  differently  scheduled  campaigns. 

Elements  of  a  complex,  long-duration  campaign  must  align  with  both  the  physical 
processes  they  will  be  studying,  and  with  the  complex  socio-political  environment  they  will 
be  implemented  in.  Complex  engineering  programs,  particularly  space  missions,  frequently 
experience  severe  cost  overruns  (GAO,  2009).  As  costs  increase,  so  does  the  likelihood  a 
program  will  be  either  descoped  or  cancelled.  For  a  campaign,  this  entails  the  cancellation 
of  later  missions.  This  was  seen,  for  example,  in  the  truncation  of  the  Apollo  program, 
where  the  original  campaign  of  20  missions  was  cut  down  to  17,  as  political  and  popular 
support  for  repeated  lunar  expeditions  waned.  Stakeholder  needs  change  over  time — 
there  is  an  incentive  on  flying  missions  earlier  in  the  campaign  sequence. 

Scheduling  also  affects  the  overlap  and  continuity  of  measurements,  it  is  often 
desirable  and  sometimes  necessary  to  have  similar  instruments  overlap  in  time  to  calibrate 
one  against  the  other.  Similarly,  it  is  sometimes  desired  to  have  a  continuous  data  record  of 
a  particular  measurement  for  use  in  physical  systems  modeling.  Additionally,  different 
data  products  can  be  produced  when  additional  measurements  are  added.  Synergistic 
effects  between  measurements  increase  the  utility  of  both  measurements  separately,  or  can 
lead  to  the  emergence  of  entirely  new  measurements. 

Finally,  a  schedule  is  affected  by  the  urgency  associated  with  each  stakeholder  need. 
Not  every  mission  must  be  accomplished  immediately.  For  the  Decadal  Survey,  this  is 
particularly  manifested  in  a  comparison  between  Climate  and  Solid  Earth  Dynamics 
missions:  the  timescales  involved  predispose  stakeholders  to  prioritize  understanding  the 
anthropogenic  changes  to  climate.  Hence,  the  future  worth  of  each  mission  is  dependent  on 
the  science  objectives  of  that  mission. 
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2.1.4  Missions 

While  scheduling  can  greatly  modify  the  transfer  of  value  to  stakeholders,  value  is 
primarily  produced  by  the  missions  in  a  campaign.  This  section  describes  different  classes 
of  purpose  attributable  to  missions  which  affect  their  expected  benefit.  Additionally,  it 
discusses  the  decomposition  of  missions  into  instruments  and  buses  as  value-related 
elements. 

The  campaign  principle  of  purpose  discussed  in  the  previous  section  manifests  itself 
into  three  classes  within  missions:  operational,  research,  and  discovery.  These 
classifications  examine  the  expected  value  of  a  particular  experiment. 

Operational  missions  are  best  described  as  components  of  long-duration  programs. 
They  are  dedicated  to  studying  a  very  specific  set  of  physical  phenomena,  and  do  so  in  a 
well  understood  manner.  These  missions  often  have  many  heritage  systems  taken  from 
their  precursors,  and  sometime  are  exact  replicas.  The  costs  incurred  by  the  mission  and 
the  benefits  accrued  are  all  very  well  understood  during  the  early  planning  phases.  The 
progression  of  value  from  mission  to  stakeholder  is  clear:  information  from  the  mission 
translates  into  useful  applications.  Operational  missions  are  usually  commissioned  based 
upon  the  proven  utility  of  their  precursors:  demonstrate  enough  benefit  to  stakeholders, 
and  continuing  to  do  so  will  be  incentivized  or  even  mandated.  A  good  example  of  an 
operational  mission  is  the  latest  LandSat.  The  LandSat  program  is  in  its  37*  year  of 
continuous  operation:  the  geospatial  information  it  provides  to  a  multitude  of  federal 
agencies,  scientists,  and  commercial  users  has  ensured  its  longevity. 

Research  missions  are  best  described  as  experiments  that  could  become 
operational.  These  missions  are  dedicated  towards  understanding  physical  phenomena  so 
that  a  useful  application  can  be  derived.  Whereas  the  focus  of  an  operational  mission  is 
very  specific,  research  missions  are  geared  towards  a  more  general  understanding.  The 
costs  and  benefits  of  a  research  mission  are  less  well  understood,  but  there  exists  a  strong 
possibility  for  unintended  benefit.  A  number  of  measurements  are  identified  a  priori  as 
potentially  valuable,  and  a  research  mission  is  designed  to  isolate  and  capture  them. 
Research  missions  are  vital  because  they  drive  the  discovery  of  useful  applications.  A  good 
example  of  a  research  mission  was  OCO,  which  was  designed  to  measure  carbon  in  the 


40 


atmosphere.  Earth  Scientists  are  currently  unable  to  close  the  carbon  loop — large  amounts 
of  the  carbon  being  sent  into  the  atmosphere  are  disappearing  somewhere.  There  are 
theories,  but  OCO  sought  to  come  up  with  a  definitive  answer  that  would  enable  more 
effective  policing  of  international  and  commercial  emissions  agreements. 

Discovery  missions  are  best  described  as  explorations  into  the  unknown.  These 
missions  are  dedicated  to  gaining  knowledge  without  any  foreknowledge  of  where  to  look; 
they  are  almost  completely  unconnected  from  specific  objectives  and  rely  upon  theory  for 
direction.  The  costs  of  these  missions  are  less  well  known,  and  there  is  almost  no  expected 
benefit.  These  missions,  however,  have  the  greatest  potential  for  unintended  benefit,  and 
serve  the  role  of  identifying  areas  for  future  research.  Earth  science  does  not  lend  itself  to 
exploration  missions:  any  undiscovered  frontier  on  the  planet  can  likely  be  explored  more 
cheaply  on  the  ground  by  people. 

These  classes  of  mission  purpose  outline  a  fundamental  spectrum  best  referred  to 
as  "mission  specificity"  (Figure  15).  This  spectrum  reflects  a  number  of  qualities  which 
describe  the  value-related  processes  of  missions.  The  scope  of  a  mission  can  be  general  or 
specific,  the  costs  can  known  or  unknown,  the  benefits  can  be  intended  or  unintended,  and 
the  goals  can  be  application  or  discovery  based.  Every  mission  falls  somewhere  on  this 
spectrum. 


Figure  15.  Spectrum  of  Mission  Specificity 

The  Decadal  Survey  actively  sought  to  strike  a  balance  of  research  missions  that 
leaned  towards  both  sides  of  the  spectrum.  While  more  operational  missions  convey 
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explicit  benefit  to  stakeholders,  discovery-focused  missions  allow  the  discovery  of  new 
applications,  and  are  essential  to  a  long-term  strategy. 

The  issue  of  intended  versus  unintended  benefit  is  perhaps  the  most  difficult 
problem  in  tracing  value.  Without  resorting  to  probabilistic  modeling,  one  solution  is 
assume  that  every  mission  in  the  Decadal  Campaign  was  far  enough  to  the  left  on  the 
spectrum  of  specificity  that  the  benefits  are  already  understood  and  can  be  quantified  and 
compared.  This  thesis  assumes  a  uniform  level  of  specificity  so  that  the  issue  of  unintended 
benefit  can  be  ignored. 

The  specific  value-related  processes  of  a  mission  can  be  decomposed  into  two 
elements:  an  instrument  and  a  bus.  As  explained  in  Sutherland’s  campaign  definition,  the 
instrument  captures  measurements  which  satisfy  objectives,  thus  providing  benefit  to 
stakeholders.  However,  as  value  is  a  function  of  both  cost  and  benefit,  it  is  necessary  to 
consider  the  costs  associated  with  buses.  For  a  space  mission,  the  bus  is  a  function  of  the 
instrument,  "the  payload  is  the  single  most  significant  driver  of  spacecraft  design"  (Larson 
&  Wertz,  1999).  A  particular  mission  can  have  one  or  many  instruments,  in  general,  there 
exists  a  correlation  between  the  number  of  instruments  on-board  a  satellite  and  the  cost  of 
the  mission,  as  the  mass  and  power  of  the  instruments  tend  to  drive  bus  costs 
parametrically. 

The  decomposition  of  missions  into  busses  and  instruments  is  not  unique  to  space 
missions — every  science  experiment  will  require  some  sort  of  support  process  that  will 
incur  costs.  A  campaign  can  be  composed  of  different  bus  types — space,  air,  and  ground 
resources  can  be  incorporated  into  the  same  framework.  For  this  thesis  only  space 
missions  are  considered. 

2.1.5  Instruments 

While  the  cost  of  a  mission  is  primarily  accrued  by  the  bus,  benefit  is  primarily 
delivered  by  the  instrument.  As  Sutherland  explained  in  his  campaign  definition, 
instruments  create  benefit  by  capturing  measurements.  Measurements  are  the  actual  data 
recorded  by  instruments  and  transmitted  back  to  earth,  and  then  interpreted  by  scientists. 
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Benefit  is  realized  through  the  data,  and  the  value  of  an  instrument  is  a  translation  of  the 
value  of  its  measurements. 

Different  instruments  capture  measurements  in  different  ways.  There  are 
substantial  quantitative  and  qualitative  differences  between  individual  instruments.  A 
small  Field  of  View  (FOV)  sensor  can  have  a  high  resolution,  and  a  low  resolution  sensor 
can  have  a  large  FOV,  but  it  is  difficult  (and  prohibitively  expensive)  to  make  a  large  FOV  hi¬ 
res  sensor.  The  effectiveness  of  a  particular  instrument  in  taking  a  particular 
measurement  in  reality  is  a  function  of  many  variables.  The  field  of  view,  coverage  gaps, 
resolution,  and  many  other  factors  dictate  how  useful  an  instrument  will  be.  For  the 
purposes  of  this  model,  these  attributes  were  condensed  into  two  attributes:  the  Quality  of 
Data  Produced,  and  the  Quantity  of  Data  Produced  (Table  2). 
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Table  2.  Utility  of  Data  Produced 


This  simplified  evaluation  metric  can  be  easily  applied  to  instruments  to  understand  their 
effectiveness  in  capturing  specific  measurements.  This  allows  for  differentiation  amongst 
instruments  that  capture  the  same  measurements. 

2.1.6  Measurements 

The  measurements  captured  by  each  instrument  convey  benefit.  While  scientists 
value  measurements  from  their  experiments,  benefit  at  this  level  is  not  differentiable:  one 
can  assume  every  scientist  finds  his  own  type  of  data  more  valuable  than  anyone  else's. 
There  is  no  architectural  significance  at  that  level;  instead,  measurements  must  be 
considered  by  the  data  products  that  can  be  derived  from  them.  Data-products  are  defined 
as  the  result  of  adding  measurements  to  practical  applications  utilized  by  large  segments  of 
society.  A  good  example  of  a  data-product  is  weather  forecasting:  the  measurement  of 
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ocean  vectors  winds  at  altitude  has  no  practical  significance  to  much  of  the  population,  but 
the  data-product  of  hurricane  landfall  predictions  does. 

At  the  time  of  mission  launch  it  is  impossible  to  predict  the  totality  of  data-products 
a  particular  mission  will  produce.  Instead,  an  ideal  reference  is  the  mission  and  program 
objectives.  These  objectives  spell  out  the  expected  data-products  as  they  relate  to 
individual  measurements  and  requirements. 

2.1.7  Objectives 

Every  campaign  can  be  expressed  as  a  series  of  objectives.  These  objectives  state 
the  intended  value:  the  practical  applications  and  uses  that  a  majority  of  society  will  benefit 
from  because  of  this  campaign.  These  objectives,  proposed  by  scientists  to  the  larger 
stakeholder  community,  indicate  the  measurements  scientists  believe  they  can  transform 
into  value. 

Objectives  have  several  key  attributes.  One  objective  may  require  several 
measurements,  and  one  measurement  could  satisfy  multiple  objectives.  An  objective  may 
have  a  primary  measurement,  which  is  essential  to  obtaining  that  objective,  and  it  may 
have  several  supporting  measurements  (which  synergize  with  the  primary).  Scientists, 
who  propose  objectives,  can  prioritize  objectives  within  their  field,  but  have  a  hard  time 
comparing  their  objectives  to  those  in  other  fields.  As  such,  the  value  of  scientific  field  is 
dependent  on  the  priorities  set  by  society. 

The  satisfaction  of  objectives  is  not  wholly  dependent  on  measurements.  Data 
containing  the  measurements  must  be  processed  and  analyzed  to  produce  the  data 
products  stakeholders  need.  This  is,  however,  an  independent  process  of  the  campaign 
architecture.  It  is  assumed  that  every  objective  will  require  some  form  of  data  processing; 
hence  this  property  of  objectives  is  architecturally  independent. 

2.1.8  Priorities 

At  its  heart,  every  campaign  is  driven  by  a  macroscopic  set  of  priorities  set  by  the 
larger  stakeholder  community.  This  stakeholder  prioritization  of  scientific  fields  is 
necessary  to  remove  the  assumed  biases  of  scientific  communities.  These  priorities  are 
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often  in  competition  for  limited  resources,  and  the  resultant  campaign  architectures  often 
reflect  that. 

The  Decadal  Survey  identified  that  such  a  consideration  was  necessary,  particularly 
in  avoiding  the  tendency  of  research  to  ignore  applications: 

Extracting  societal  benefit  from  space-based  measurements  requires,  as  an  equally 
important  second  step,  the  development  of  a  strong  linkage  between  the 
measurements  and  the  decision  makers  who  will  use  them.  This  linkage  must  be 
created  and  sustained  throughout  the  life  cycle  of  the  space  mission.  In  implementing 
future  missions,  scientists  engaged  in  research  intended  to  make  both  scientific  and 
societal  contributions  must  operate  differently  than  they  did  when  the  advancement  of 
science  was  the  primary  or  only  goal  of  research.  (National  Research  Council,  2007) 

The  linkages  between  measurements  and  stakeholders  must  be  a  consideration  of  any 

campaign  design,  and  the  prioritization  of  different  communities  of  science  is  the  first  step 

in  establishing  these  linkages.  Frequently  these  prioritizations  are  made  evident  in  high- 

level  policy  documents.  Weiss’  paper  on  Science  Traceability  identifies  program  objectives, 

NASA  roadmaps,  and  Academy  of  Science  surveys,  as  key  sources  of  this  information 

(Weiss,  Smythe,  &  Lu,  2005). 

2.1.9  Overview  of  Value  Decomposition 

The  decomposition  of  a  science  campaign  outlines  how  value  traces  through  the 
system.  First,  stakeholder  set  priorities.  These  priorities  are  then  more  formally  codified 
as  science  objectives.  The  objectives  require  a  specific  set  of  measurements  to  produce 
valuable  data  products.  Measurements  are  captured  by  instruments,  which,  along  with  the 
bus  elements,  define  the  missions.  The  combination  of  a  set  of  missions  with  a  schedule 
defines  a  campaign.  Both  benefits  and  costs  accrued  in  this  framework  can  be  traced  to 
their  sources,  as  every  element  in  the  decomposition  is  considered  architecturally 
significant. 
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2.2  A  Framework  for  Campaign  Analysis:  the  CSTM 


The  value  decomposition  presented  in  the  previous  section  outlines  the  key 
elements  of  a  campaign  and  enables  value  traceability.  This  section  introduces  a 
methodology  for  keeping  track  of  these  elements  and  the  relationships  between  them  a 
large  number  of  missions  over  time,  the  Campaign  Science  Traceability  Matrix. 


The  value-decomposition  framework  of  the  previous  is  advantageous  because  it 
allows  the  traceability  of  value  in  a  campaign,  it  does  not,  however,  express  specific 
relationships.  The  Science  Traceability  Matrix  described  by  Weiss  is  contrastingly 
advantageous  because  it  succinctly  relates  different  elements  and  requirements  of  a 
mission,  it,  however,  is  only  designed  to  describe  a  single  mission.  A  new  framework  was 
developed  to  incorporate  the  advantages  of  both  frameworks  at  a  campaign  level:  the 
Campaign-level  Science  Traceability  Matrix  (Figure  16). 


Feasible 
Campaign 
Architectures  IT 


Figure  16.  The  Campaign-Level  Science  Traceability  Matrix  (CSTM) 
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The  CSTM  expresses  different  levels  of  campaign  decomposition  as  a  series  of 
matrices.  Each  matrix  expresses  to  the  qualitative  relationships  between  multiple 
instances  of  each  campaign  element.  Hence,  relationships  such  as  "one  objective  requiring 
many  measurements"  can  be  expressed  by  populating  a  matrix — measurements  on  one 
axis  and  objectives  on  the  other.  While  these  are  expressed  as  matrices,  matrix  math  does 
not  necessarily  apply,  and  relationships  can  be  described  as  unique  functions.  This  allows 
for  the  architecting  of  a  campaign  on  multiple  levels: 

•  The  assignment  of  dates  to  a  particular  mission  set  (scheduling) 

•  The  assignment  of  missions  to  a  particular  mission  set  (determining  the 
mission  content) 

•  The  assignment  of  instruments  to  missions  (mission  design) 

Similarly,  the  impacts  of  architectural  decisions  can  be  traced  on  multiple  levels: 

•  The  benefit  accrued  by  capturing  measurements 

•  The  benefit  accrued  by  satisfying  objectives 

•  The  benefit  accrued  by  contributing  to  priorities 

This  framework  allows  the  architect  to  take  into  consideration  many  different  levels  of 
information,  and  judge  the  actual  benefits  associated  with  his  decisions.  It  provides  the 
basis  for  campaign  cost-benefit  analysis  not  only  in  assembling  his  missions,  but  also  in 
determining  which  missions  to  fly  and  when  to  fly  them. 

The  CSTM  can  also  be  expressed  more  generally  in  terms  of  the  flow  knowledge 
through  the  system  (Figure  17).  Societal  concerns  and  stakeholder  needs  form  the 
foundation  of  value  discussions.  Then  this  information  must  interpreted  by  policy-minded 
scientists,  such  as  the  Decadal  Panel,  to  provide  concrete  priorities  and  objectives.  Third, 
scientific  knowledge  must  be  applied  to  the  specific  implementation  of  these  objectives, 
particularly  in  the  design  of  instruments.  Informed  by  that  discussion,  engineering 
knowledge  is  then  required  to  determine  the  proper  manifesting  of  instruments  to 
missions,  and  the  scheduling  of  those  missions.  Finally,  the  cumulative  knowledge  implicit 
in  the  campaign  analysis  informs  the  system  architect  the  optimal  manner  in  which  to  plan 
his  campaign. 
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Figure  17.  Generalized  CSTM 

The  CSTM  is  tool  for  envisioning  the  traceability  of  value  in  a  campaign.  By 
decomposing  a  campaign  into  progressively  smaller  pieces,  the  architecturally  significant 
components  can  be  isolated.  By  representing  the  relationships  between  hierarchical 
components  as  mappings,  the  impacts  of  architectural  decisions  can  be  traced  to  every 
other  component.  The  key  advantage  this  method  delivers  is  flexibility:  changes  can  be 
made  on  any  level,  and  the  effects  can  be  easily  traced. 
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3  Applying  the  CSTM  Framework  to  the  Decadal  Survey  Campaign 


The  CSTM  framework  described  in  the  previous  chapter  is  a  tool  that  enables 
traceability  of  campaign  design  decisions  to  stakeholder  impacts.  It  is  useful  in  logically 
enumerating  the  relationships  between  different  elements  of  a  campaign.  The  impacts  of 
system  architecting,  both  on  the  campaign  schedule  and  individual  mission  level,  can  be 
analyzed  as  they  affect  the  delivery  of  value  to  stakeholders  through  priorities  and 
objectives. 

This  chapter  describes  in  detail  the  application  of  the  CSTM  to  the  Decadal 
campaign.  The  objectives  of  this  chapter  are: 

Objectives  of  applying  the  CSTM  to  the  Decadal  Survey 

•  To  apply  the  framework  for  analysis  to  the  Decadal  campaign  by  enumerating  the 
relationships  between  campaign  elements  using  stakeholder  modeling,  the  Decadal 
Survey,  and  NASA  surveys 

•  To  validate  the  methods  and  model  introduced  in  this  thesis  by  comparing  results 
using  the  Decadal  Survey  as  “truth" 

•  To  examine  the  architectural  impacts  of  value  on  campaign  design  by  tracing 
science  value  using  the  CSTM 

This  chapter  outlines  the  specific  methodology  used  to  populate  the  CSTM  for  the 
Decadal  Survey.  While  the  Decadal  Survey  included  a  brief  discussion  of  ground  and  air 
campaigns  and  introduced  the  concept  of  Venture-class  small  satellites  to  further  complete 
its  science  objectives,  this  chapter  only  focuses  on  the  incorporation  of  the  17  named 
missions  to  the  CSTM.  This  chapter  is  organized  into  the  following  sections: 

•  Section  3.1:  Populating  the  CSTM.  This  section  presents  the  methodology  used 
to  populate  the  different  elements  of  the  CSTM. 

•  Section  3.2:  Comparison  to  the  Decadal  Survey.  This  section  compares  the 
completed  model  to  the  Decadal  Survey. 

•  Section  3.3:  Examination  of  Science  Traceability.  This  section  examines  the 
scientific  decisions  made  by  the  Decadal  Survey  with  respect  to  stakeholders. 

•  Section  3.4:  Summary  of  the  CSTM. 
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3.1  Populating  the  CSTM 


The  Decadal  Survey  proposed  a  complete  campaign:  starting  with  priorities  one 
could  trace  the  logical  connections  through  the  CSTM  and  out  to  the  notional  launch  tiers  of 
specific  missions  (Figure  16).  The  specific  nature  of  these  relationships,  however,  was  not 
uniformly  presented  in  a  clear  and  recoverable  manner,  and  they  largely  lacked  qualitative 
assertions.  Using  a  combined  approach  of  the  Decadal  Survey,  Sutherland's  stakeholder 
analysis,  and  a  survey  of  NASA  scientists  and  engineers,  the  CSTM  matrices  were 
populated. 


3.1.1  Representing  the  Six  Decadal  Science  Panels  as  Priorities 

The  first  step  undertaken  in  applying  the  CSTM  to  the  Decadal  Survey  was 
quantifying  stakeholder  priorities  (represented  in  the  green  lower-right  box  of  Figure  16). 
One  of  the  outputs  of  Sutherland's  stakeholder  analysis  of  the  Earth  Observation  Program 
was  a  relative  comparison  of  the  value  different  Decadal  Survey  science  panels  with 
regards  to  the  stakeholder  network  (Figure  18). 
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Figure  18.  Sutherland's  Stakeholder  Analysis  of  Panel  Weighing 


(Sutherland,  2009) 
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These  six  panels  are  sufficiently  representative  of  stakeholder  priorities  for  use  in 
the  CSTM.  According  to  Sutherland,  the  most  valuable  science  community  to  the  Earth 
Observation  enterprise’s  stakeholders  is  represented  by  the  weather  panel:  weather  data- 
products  are  utilized  extensively  by  millions  of  people  on  a  daily  basis.  The  Climate-change 
and  Land-use  panels  are  tied  as  the  second  most  valuable,  reflecting  the  large  segment  of 
shareholders  that  utilize  geo-spatial  information,  as  well  as  the  looming  societal  issue  of 
anthropogenic  climate  change.  The  Water  panel,  although  ranked  fourth,  is  of  median 
importance,  reflective  of  the  growing  awareness  of  water  as  a  limited  resource.  The  lowest 
scoring  panels.  Human  Health  and  Security  and  Solid  Earth,  are  explained  by  Sutherland  as 
non-traditional  priorities  for  NASA  and  NOAA.  Using  Sutherland’s  valuations,  a  relative 
weighting  of  each  panel  can  be  accomplished  (Equation  1). 

Occurance  j 

Normal  j  ^6 

2ji=i  ^ p  Occurance  ; 

Equation  1.  Normalized  Panel  Weighting 

The  weighted  occurrence  score  of  each  panel  can  be  normalized  to  the  relevant  fraction  of 
the  total  benefit  in  the  system  (Table  3): 


Science  Panel 

Weighted  Occurrence  Score 

Normalized  Fraction  of  total  Benefit 

Weather 

8.65 

0.214 

Climate  change 

8.33 

0.206 

Land-use 

8.33 

0.206 

Water 

6.31 

0.156 

Human  health 

4.49 

0.111 

Solid  Earth 

4.31 

0.107 

Total 

40.42 

1.000 

Table  3.  Normalized  Panel  Weights 


The  normalized  fraction  of  total  benefit  is  a  weighing  that  can  be  found  for  every 
panel  with  regards  to  the  totality  of  benefit  in  the  system.  It  is  assumed  that  the  17 
missions  of  the  Decadal  Survey  campaign  will  produce  100%  of  the  possible  benefit. 
Synergistic  effects  are  ignored  in  this  initial  computation,  as  Decadal  Survey  panelists  were 
instructed  to  select  missions  based  on  the  assumption  of  isolation  (every  mission  is  a 
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stand-alone).  The  stakeholder  prioritizations  of  the  Decadal  Survey  science  panels  can 
then  be  utilized  to  weight  each  community’s  objectives. 

3.1.2  Utilizing  the  Panel  Objectives  from  the  Decadal  Survey 

Each  of  the  science-themed  Decadal  Survey  panels  outlined  a  set  of  prioritized 
objectives  for  the  campaign.  These  can  be  scaled  and  incorporated  into  the  lower  blue 
matrix  of  CSTM  in  Figure  16  to  relate  stakeholder  priorities  to  measurements. 

Additionally,  the  normalized  weighting  of  the  panels  can  be  applied  to  the  panel-ranked 
objectives  to  create  an  absolute  prioritization  of  the  objectives. 

The  Decadal  Survey  Committee  was  given  the  set  of  tasks  depicted  in  Figure  1 
(Section  1.2).  This  included  instructions  to,  "develop  a  consensus  of  the  top-level  scientific 
questions  that  should  the  focus  for  earth  and  environmental  observations"  and, 
"recommend  a  prioritized  list  of  measurements".  These  tasks  were  given  to  the  different 
panels,  and  the  specific  implementation  varied  significantly.  However,  one  process  almost 
universally  followed  was  the  creation  of  prioritized  objectives  which  would  answer  each 
panel's  top-level  questions.  The  following  guidance  was  provided  to  individual  panels: 


BOX  5.2  QUESTIONS  FOR  PLANNERS  TO  USE  IN  INCORPORATING  APPLICATIONS 
WHEN  SETTING  PRIORITIES  FOR  MISSION  SELECTION 


•  What  is  the  immediate  need?  What  is  the  projected  need? 

•  Hasan  analysis  ofbenefits  been  done?  Who  are  the  beneficiaries?  How  does  information  from  measure¬ 
ments  reach  them? 

•  What  alternative  sources  of  information  exist  for  the  application?  In  situ  sources?  Foreign  sources?  Is 
the  proposed  measurement  or  mission  a  demonstrable  improvement? 

•  To  what  degree  does  the  measurement  need  to  be  operational  or  continuous?  Can  it  be  a  periodic  or 
a  one-time  measurement? 

•  What  are  the  requirements  for  timeliness  in  delivery  of  products? 

•  What  are  the  means  for  funneling  data  to  decision  makers,  either  directly  or  indirectly  through  data 
brokers  (for  example,  the  Weather  channel)  or  interpreters  (such  as  nongovernmental  organizations)? 
What  is  the  commitment  on  their  part  to  use  the  data? 

•  What  are  the  necessary  ancillary  data?  How  are  they  to  be  made  available? 

•  Are  necessary  simulation,  analytic,  or  visualization  tools  in  place? 

•  What  is  the  weakest  link  in  the  chain  from  measurement  to  use? 

•  What  are  the  risks  if  the  measurement  is  not  made? 


Figure  19.  Prioritizing  Objectives  (National  Research  Council,  2007) 
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In  answering  these  questions,  the  panels  enumerated  a  set  of  specific  objectives  to 
be  accomplished  in  the  decade.  These  objectives,  and  their  corresponding  specific 
measurements,  instrument  types  and  basic  requirements,  and  mission  implementations, 
were  summarized  in  tables  such  as  seen  in  Figure  20.  With  the  exception  of  the  Human 
Health  and  Security,  every  panel  prioritized  their  objectives. 


TABLE  7.1  Land-Use  Change  and  Ecosystem  Dynamics  Panel  Priority  New  Missions 


Summary  of 
Mission  Focus 

Variables 

Ecosystem 

TerrestriaL 

function: 

Distribution 

climate  and 

and  changes 

land -use 

in  key  species 

impacts  on 

and  functional 

terrestrial 

groups  of 

and  coastal 

organisms. 

ecosystems 

disturbance 
patterns, 
vegetation 
stress,  vegetation 
nutrient 
status,  primary 
productivity, 
vegetation  cover 
Coastak 

coral-reef  health 
and  extent 

Ecosystem 

Standing 

structure  and 

biomass, 

biomass 

vegetation 
height  and 
canopy  structure, 
habitat  structure 

Carbon  budget 

CO2  mixing 
ratio,  CO 
concentrations 

Type  of  Sensor  Coverage 


Spatial 

Resolution 


Frequency 


Synergies 
with  Other 
Panels 


Related  Planned 
or  Integrated 
Missions 


Hyperspectral 


Global 

pointable 


50-75  m  30  day,  Climate 

pointable  to  Health 
daily  Solid  Earth 


Lidarand  InSAR  Global 


50- 150  m  Monthly 


Active  lidar 


Global 


100  m  strips  Diurnal — 
assimilated 
every  24 
hours 


HyspIRI 


Climate 
Health 
Solid  Earth 


Climate 

Weather 


DESDynl 

ICESat-ll 


ASCENDS 


Figure  20.  Land-Use  and  Ecosystems  Objectives  (National  Research  Council,  2007) 


In  this  format,  the  left  hand  column  provides  a  brief  descriptor,  the  summary  of  mission 
focus,  describing  the  specific  objective.  Subsequent  columns  provide  the  specific 
requirements  needed  to  achieve  this  objective  according  to  the  planned  implementation. 
Objectives  are  presented  in  prioritized  order. 

For  inclusion  in  the  CSTM,  a  majority  of  the  objectives  enumerated  in  the  Decadal 
Survey  were  left  untouched.  Table  4  enumerates  the  mapping  of  CSTM  objectives  to 
"Mission/Observation  Type"  identified  in  Table  2.3  of  the  Decadal  Survey,  with  the  Decadal 
represented  on  the  vertical  axis  and  the  CSTM  represented  on  the  horizontal  axis. 
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Ecosystem  fimction 

X 

Ecosystem  strnctme 
andbiomass 

X 

CadKmbD^et 

X 

(Hobal  ecosystem 
dynamics 

X 

Global  ocean 
pioAictivity 

X 

Solid  Earth 

Surfacadeformatlon 

Surface  composition  and 
thermal  Dr obartles 

High  resolution 
topography 

Temporal  variations  In 
Earth'  sgravltv  field 

> 

1 

> 

JZ 

1 

Sur&ce  deformation 

X 

Snrfoce  composition 
and  thermal  properties 

X 

H^-resohitioo 

X 

Table  4.  Objective  Identification 
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A  small  number  of  objectives  were  broken  apart  into  more  atomic  pieces  when  no 
rationale  was  provided  for  the  convolution  of  apparently  disparate  sub-objectives,  as  seen 
in  the  climate  objective  "Clouds,  Aerosols,  ice,  and  Carbon".  Additionally,  the  Decadal 
summary  of  "Mission/Observation  Types"  did  not  include  two  of  the  Solid  earth  objectives, 
which  were  added  to  the  CSTM  list.  After  correction,  a  total  of  37  Objectives  identified  in 
Table  4  were  added  to  the  CSTM. 

While  the  Decadal  Survey  panel  reports  provide  the  prioritizations  amongst  these 
objectives,  they  do  not  explicit  quantify  how  much  more  important  one  is  over  another. 
Hence,  a  subjectively-tuned  scaling  algorithm  was  necessary  to  translate  the  language  of 
the  Decadal  Survey  into  a  computationally  useful  metric. 

First,  it  was  assumed  that  the  median  ranked  objective  would  have  the  mean 
objective  weighting  (Equation  2). 

_  1 

^int  b  median  y 

i  is  the  number  of  objectives  for  panel  J 

Equation  2.  Median  Objective  Weight 


This  intermediate  (unnormalized)  weighting  was  assigned  to  the  median  objective.  Hence, 
if  a  panel  had  five  prioritized  objectives,  the  third  would  be  assigned  an  un-normalized 
weighting  of  -  or  0.2.  The  other  objectives  were  weighed  linearly  with  regards  to  the 
median  based  upon  a  subjective  slope,  set  by  varying  the  z-values  for  the  panel: 

median  +2  (y^int  b  median  ”F  2z) 

^b  median  +1  (Wint  b  median  ”F  z) 


^b  median  (y^int  b  median  )(1) 

^b  median  —1  (y^int  b  median  )(l-z) 
^b  median  —2  (y^int  b  median  )(l-2z) 


Equation  3.  Linear  Scaling  around  median-mean 
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Because  some  panels  gave  the  same  ranking  to  multiple  objectives,  care  had  to  be  taken 
such  that  the  linearization  was  maintained  and  the  sum  of  the  objective  weights  equaled 
one.  Slopes  (z-values)  were  modified  based  upon  descriptions  given  in  the  Decadal  Survey 

panel  chapters,  although  the  default  used  was  z=0.25.  As  the  Human  Health  and  Security 

1 

panel  did  not  prioritize  objectives,  every  objective  was  weighted  equally  at  -,  where  i  —  7. 

Additionally,  the  Water  panel  had  two  linearizations,  based  upon  having  two  distinct  tiers 
of  objectives.  The  weights  assigned  to  every  objective  are  listed  in  Table  5  below. 


Panel 

Objective 

Rank 

within 

panel 

Normalized 

Weighting 

Panel 

weight 

Absolute 
Objective  Weight 

Ozone  Processes:  Ultraviolet  Radiation  and  Cancer 

1 

0.17 

0.019 

Heat  stress  and  drought 

1 

0.17 

0.019 

Acute  Toxic  Pollution  Releases 

1 

0.17 

0.019 

<T3 

0.111 

X 

Air  Pollution  and  Respiratory  and  Cardiovascular  Diseases 

1 

0.17 

0.019 

Algal  Blooms  and  Waterborne  Infectious  Diseases 

1 

0.17 

0.019 

Vector-borne  and  Zoonotic  Disease 

1 

0.17 

0.019 

to 

Ecosystem  Function 

1 

0.28 

0.058 

£ 

0) 

Ecosystem  Structure  and  Biomass 

2 

0.24 

0.049 

VI 

>• 

Carbon  Budget 

3 

0.20 

0.206 

0.041 

o 

u 

Coastal  Ecosystem  Dynamics 

4 

0.16 

0.033 

Global  Ocean  Productivity 

5 

0.12 

0.025 

Surface  deformation 

1 

0.29 

0.031 

t 

Surface  composition  and  thermal  properties 

2 

0.24 

0.025 

LU 

High  resolution  topography 

3 

0.19 

0.107 

0.020 

o 

Temporal  variations  in  Earth's  gravity  field 

4 

0.14 

0.015 

Oceanic  bathymetry 

4 

0.14 

0.015 

Aerosol-Cloud  Forcing 

1 

0.18 

0.037 

Ice  Sheet  and  Sea  Ice  Volume 

1 

0.18 

0.037 

01 

Carbon  Sources  and  Sinks 

1 

0.18 

0.037 

E 

Radiance  Calibration  and  Time-Reference  Observatory 

2 

0.14 

0.206 

0.029 

u 

Earth  Radiation  Budget  (ERB)  Continuity 

2 

0.14 

0.029 

Ice  Dynamics 

3 

0.11 

0.022 

Ocean  Circulation,  Heat  Storage,  and  Climate  Forcing 

4 

0.07 

0.015 

Tropospheric  winds 

1 

0.19 

0.041 

High-Temporal-Resolution  Air  Pollution 

1 

0.19 

0.041 

All-Weather  Temperature  and  Humidity  Profiles 

2 

0.15 

0.033 

<T3 

Comprehensive  Tropospheric  Aerosol  Characterization 

2 

0.15 

0.214 

0.033 

Radio  Occultation 

3 

0.12 

0.025 

Comprehensive  Tropospheric  Ozone  Measurements 

3 

0.12 

0.025 

Aerosol-Cloud  Discovery 

4 

0.08 

0.016 

Soil  Moisture  and  Freeze-Thaw  State 

1 

0.29 

0.045 

Surface  Water  and  Ocean  Topography 

2 

0.24 

0.037 

Snow  and  Cold  Land  Processes 

3 

0.19 

0.030 

<T3 

Water  Vapor  Transport 

4 

0.10 

0.156 

0.015 

5 

Sea  Ice  Thickness,  Glacier  Surface  Elevation,  and  Glacier  Velocity 

5 

0.08 

0.012 

Groundwater  Storage,  Ice  Sheet  Mass  Balance,  and  Ocean  Mass 

6 

0.06 

0.010 

Inland  and  Coastal  Water  Quality 

7 

0.05 

0.007 

Total 

1.000 

Table  5.  Objective  Weighting 
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This  methodology  introduces  some  artifacts.  Panels  with  a  large  number  of 
objectives  are  penalized,  as  the  median  weighted  objective  is  valued  as  the  mean.  This  is 
not  an  inherently  incorrect  assumption,  as  it  mimics  human  thought  processes.  It  is 
possible  the  Climate  panel  intentionally  convoluted  their  objectives  to  avoid  this  bias. 

Although  objectives  are  weighed  within  each  panel,  the  normalized  panel  weights 
can  be  applied  to  each  objective  to  produce  an  absolute  measure  of  benefit  for  each 
objective  (Equation  4). 

absolute  )  *  (y^panel  7) 

Equation  4.  Absolutely  Weighted  Objectives 

The  absolute  weight  of  the  objective  is  equal  to  the  product  of  the  panel  weight  with  the 
objective  weight  (Table  5).  Objectives  are  assumed  to  be  unique  to  panels,  so  that  the  sum 
of  every  absolutely  weighted  objective  from  a  particular  panel  equates  to  the  normalized 
panel  weight.  Expressed  in  terms  of  value  traceability,  the  benefit  to  society  of  a  particular 
science  community  is  completely  divided  among  its  objectives.  Hence  satisfying  each  of 
those  objectives  will  contribute  that  panel's  value  to  the  enterprise  stakeholders. 
Additionally,  since  the  panel  weights  were  normalized  as  well,  the  sum  of  every  objective 
across  all  panels  will  equal  one.  The  Decadal  Survey  is  "complete,"  if  every  objective  is 
satisfied,  100%  of  the  value  in  the  system  will  be  delivered. 

These  weighting  can  alternately  be  plotted  by  objective  (Figure  21).  The  most 
beneficial  objective  is  "Ecosystem  function"  (the  top  objective  from  the  second-most 
important  panel).  Although  the  Weather  panel  was  weighted  the  highest,  because  it 
proposed  7  objectives,  the  value  of  each  was  comparably  less  than  the  Land-use  and 
Ecosystems  panel,  which  only  proposed  5.  The  least  beneficial  objective  is  "Inland  and 
coastal  water  quality"  (the  last  objective  of  the  #4  panel).  This  is  reflective  of  the  water 
panel  having  two  distinct  linearizations,  one  necessarily  lower  than  the  other. 
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Figure  21.  Absolutely  Weighted  Objectives 
3.1.3  Derivation  of  Measurements 

Each  objective  can  be  described  in  terms  of  the  measurements  required  for 
objective  satisfaction.  While  not  always  explicitly  enumerated,  the  relevant  measurements 
were  recoverable  from  the  Decadal  Survey.  A  common  set  of  measurements  was  derived 
and  mapped  to  the  CSTM  objectives.  This  section  discusses  the  population  of  the  higher- 
right  blue  box  of  the  CSTM. 

While  one  of  the  stated  tasks  of  the  Decadal  Survey  was  to  "recommend  a  prioritized 
list  of  measurements,"  this  was  not  explicitly  done  (Figure  1).  Only  two  panels  directly 
listed  the  measurements  they  required;  however,  every  panel  described  the  measurements 
relevant  to  their  ranked  objectives.  The  primary  source  of  this  information  is  the 
paragraph  descriptions  of  each  objective. 

While  the  CSTM's  matrix  format  is  more  expressive  than  the  Science  Traceability 
Matrix,  it  was  still  desirable  to  limit  the  number  of  measurements  enumerated  so  that 
instruments  of  similar  capability  could  realistically  be  compared  against  each  other.  To 
this  end,  a  common  set  of  measurements  was  created  to  which  all  the  objectives  in  the 
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Decadal  Survey  could  be  mapped.  It  was  also  desirable  that  this  set  of  measurements  be 
easily  traceable  to  other  pre-existing  and  international  missions;  hence,  the  Committee  on 
Earth  Observation  Satellites  (CEOS)  database  categorization  scheme  was  chosen  to  form 
the  backbone  of  the  common  set  of  measurements,  and  the  Decadal  Survey  was  used  to  fill 
in  the  details. 

in  the  CEOS  database  classification  scheme,  earth  science  is  divided  into  five 
primary  science  areas  (Table  6).  Unlike  the  six  Decadal  Survey  panels,  these  fields  are  less 
application-based  and  are  more  reflective  of  Bretherton’s  earth  system  (Figure  8).  The 
CEOS  database  outlines  27  measurement  types  within  these  five  fields.  A  28* 
measurement  type,  "Surface  water  distribution"  was  added  to  the  CEOS  list  from  the 
Decadal  Survey.  The  Surface  Water  and  Ocean  Topography  mission’s  (SWOT)  inclusion  in 
the  Decadal  campaign  indicated  this  type  of  measurement  was  important,  yet  it  did  not 
readily  lend  itself  to  inclusion  in  any  other  measurement  type,  so  a  new  category  was 
added. 


CEOS  Science  Fields 

Measurement  Types 

1)  Atmosphere 

1.1.  Aerosol  properties 

1.2.  Atmospheric  temperature  fields 

1.3.  Water  vapor 

1.4.  Atmospheric  winds 

1.5.  Cloud  type,  amount,  and  top  temp. 

1.6.  Cloud  particle  properties  and  profile 

1.7.  Liquid  water  and  precipitation  rate 

1.8.  Ozone 

1.9.  Radiation  budget 

1.10.  Trace  gases  [excluding  ozone] 

2) Land 

2.1.  Albedo  and  reflectance 

2.2.  Land  topography 

2.3.  Soil  moisture 

2.4.  Vegetation 

2.5.  Surface  temperature  [land] 

2.6.  Multi-purpose  imagery  [land] 

2.7  Surface  water  distribution* 

3)  Ocean 

3.1.  Ocean  colour/biology 

3.2.  Ocean  topography/currents 

3.3.  Ocean  salinity 

3.4.  Ocean  surface  winds 

3.5.  Surface  temperature  [ocean] 

3.6.  Ocean  wave  height  and  spectrum 

3.7.  Multi-purpose  imagery  [ocean] 

4)  Snow  and  Ice 

4.1.  Ice  sheet  topography 

4.2.  Snow  cover,  edge  and  depth 

4.3.  Sea  ice  cover,  edge  and  thickness 

5)  Gravity  and  Magnetic  fields 

5.1.  Gravity,  magnetic  and  geodynamic 

Table  6.  CEOS  Science  Fields  and  Measurement  Types 
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These  categorizations,  however,  were  determined  to  be  too  generic  to  be 
architecturally  distinguishing.  For  example,  the  measurement  type  "Aerosol  Properties" 
can  be  described  through  numerous  measurements,  such  as  height,  composition,  scattering 
properties,  size,  and  distribution,  many  of  which  require  different  instrumentation. 

Specific  measurements  were  then  added  to  this  hierarchy  based  upon  information 
in  the  Decadal  Survey.  Using  the  "variable"  information  in  the  panel  priority  tables  (Figure 
20)  and  the  descriptions  of  each  objective,  a  list  of  specific  measurements  was  derived 
(Table  7).  This  list  was  then  screened  for  duplicates  and  sufficiently  similar  measurements 
to  down-select  to  81  measurements.  These  were  then  placed  into  the  28  measurements 
types  categories  to  complete  the  common  set  of  measurements. 

The  mapping  of  specific  measurements  to  measurement  categories  can  be  found  on 
the  horizontal  axis  of  Table  7.  Additionally,  this  chart  maps  the  relationships  between  the 
81  measurements  and  the  37  objectives. 
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Table  7.  Objective  to  Measurement  Mapping 

This  mapping  was  accomplished  by  reviewing  the  descriptions  of  objectives  for 
relationships.  For  example,  the  following  four  measurements  were  derived  from  the 
descriptions  of  the  Carbon  Budget  objective  for  the  Land-use  and  Ecosystems  panel: 
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Example  Objective — Carbon  Budget 

1.  "Although  the  OCO  will  yield  a  vastly  increased  volume  of  data  for  characterizing 
the  distribution  of  atmospheric  C02  and  inferring  surface  sources  and  sinks, 
unavoidable  physical  limitations  are  imposed  by  the  passive-measurement 
approach,  including  daytime  and  high-Sun-only  sampling" 

2.  "The  first  step  in  inferring  terrestrial  ecosystem  processes  from  atmospheric 
data  is  to  separate  photosynthesis  and  respiration;  for  this,  diurnal  sampling  is 
required  to  observe  nighttime  concentrations  resulting  from  respiration  " 

3.  "it  is  also  essential  to  separate  physiological  fluxes  from  biomass  burning  and 
fossil-fuel  combustion,  and  this  requires  quasisimultaneous  measurement  of 
an  additional  tracer,  ideally  CO" 

4.  "A  laser  sounder  mission,  consisting  of  simultaneous  laser  remote  sensing  of  C02 
and  02  (needed  to  correct  for  atmospheric  pressure,  topography,  and 
target-height  effects)  would  provide  new  active  measurement  capabilities  to 
overcome  the  most  serious  of  those  limitations" 

Figure  22.  Example  Text  Descriptions  of  Objective-Measurement  Mapping  (National 

Research  Council,  2007) 

in  this  example,  the  text  reveals  two  alternate  primary  measurements  that  could 
satisfy  this  objective:  vertically  resolved  C02  measured  at  day,  and  vertically  resolved  C02 
measured  at  day  and  at  night.  This  objective  could  be  satisfied  using  existing  technology, 
such  as  the  daytime  sensor  on  OCO  (had  OCO  not  failed  in  launch),  or  it  could  be  satisfied 
using  the  newer,  less  mature  instrument  proposed  for  the  ASCENDS  mission.  For  this 
mapping,  no  distinction  is  made  between  the  qualities  of  the  two  methods,  since  both 
would  contribute  to  the  objective.  Additionally,  the  text  reveals  that  two  complimentary 
measurements  would  be  beneficial  in  satisfying  this  objective:  CO  and  02  concentrations. 
Although  in  reality  these  measurements  are  not  essential,  for  the  CSTM,  measurements 
were  mapped  to  objectives  in  binary:  either  they  did  contribute  or  they  did  not  contribute 
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to  the  satisfaction  of  an  objective.  There  was  insufficient  detail  in  the  Decadal  Survey  to 
determine  qualitative  differences. 


Example  Objective — Carbon  Budget 

1.10.8  vertically  resolved  C02  (Daytime  only) 

X 

1.10.9  vertically  resolved  C02  (Day/Night) 

X 

1.10.11  CO  concentrations 

X 

1.10.12  02  concentrations 

X 

Table  8.  Example  Objective-Measurement  Mapping 


Several  assumptions  are  implicit  in  the  mapping  from  objectives  to  measurements 
in  the  CSTM.  First,  no  distinction  was  made  between  essential  and  complimentary 
measurements  within  an  objective.  Second,  one  objective  could  be  satisfied  by  multiple 
measurements  (the  lowest  was  two,  the  highest  was  thirteen).  Finally,  one  measurement 
could  contribute  to  the  satisfaction  of  multiple  objectives. 

Having  achieved  a  mapping  of  objectives  to  measurements,  it  became  possible  to 
apply  the  absolute  weighting  of  objectives  to  the  measurements.  The  matrix  representing 
the  mapping  in  Table  7  is  a  binary  matrix  with  values  of  zero  and  one  only,  of  the 
dimension  37x81  (objectives  x  measurements),  and  is  referred  to  as  M.  This  matrix  is  first 
normalized  by  the  number  of  measurements  per  objective  (Equation  5),  to  weight 
measurements  equally  within  an  objective.  It  is  then  multiplied  by  the  absolutely  weighted 
objectives  (Equation  6).  Finally,  the  weighted  measurements  mappings  are  summed 
across  all  objectives  (Equation  7)  to  compile  to  absolutely  weighted  measurements. 

1 

normalized 

Ii  rrifc 

Equation  5.  Normalized  M 

^m,b  (j^b  normalized  absolute  ) 

Equation  6.  Weighted  M 
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37 


absolute  ^m.b 

b=l 


Equation  7.  Absolutely  Weighted  Measurements 


A  discussion  of  the  resultant  absolutely  weighted  measurements  can  be  found  in  Section 

3.3. 

Finally,  although  no  distinction  was  made  within  an  objective  as  to  the  relative 
importance  of  measurements,  the  relationships  between  measurements  could  be  evaluated 
in  an  attempt  to  quantify  synergistic  effects.  Just  as  multiple  measurements  are  sometimes 
necessary  to  fulfill  a  particular  objective,  the  presence  or  absence  of  one  measurement 
fundamentally  affects  the  utility  of  another.  One  area  for  future  work  is  to  quantify  the 
synergies  between  measurements. 


3.1.4  Instrument  utility  as  evaluated  by  NASA 

The  value  of  stakeholders  priorities  have  been  traced  to  individual  measurements. 
Every  instrument  proposed  in  the  Decadal  Survey  captures  the  measurements  necessary 
for  objective  satisfaction;  however,  the  Decadal  Survey  offers  no  clues  as  to  which 
instruments  within  a  particular  mission  capture  which  measurement,  and  provides  very 
little  information  as  to  how  effective  the  proposed  instruments  are.  A  survey  of 
instrument-measurement  relationships  was  given  to  NASA  scientists  and  engineers  to 
better  capture  these  relationships.  This  information  was  used  to  populate  the  central 
purple  box  of  the  CSTM  (Figure  16). 

The  instruments  contained  in  the  17  Decadal  missions  were  isolated  and  evaluated 
with  respect  to  the  common  set  of  measurements  (Appendix  A:  NASA  Worksheet 
Instructions).  This  evaluation  attempted  to  capture  the  qualitative  and  quantitative 
differences  amongst  instruments  as  simplified  in  Table  2,  and  was  evaluated  in  survey  form 
by  NASA  earth  scientists.  Although  the  scoring  was  done  with  integers,  NASA  responses 
were  converted  to  an  exponential  score  (Table  9) 
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Survey 

Score 

Scaled 

Scored 

0 

0 

1 

0.1 

2 

0.2 

3 

0.4 

4 

0.8 

Table  9.  Exponential  Scaling  for  Instrument-Measurement  Scores 

Using  the  instrument-measurement  scores  provided  by  NASA,  it  was  possible  to 
calculate  how  well  each  instrument  satisfied  each  objective.  Although  several  options 
presented  themselves  for  determining  satisfaction,  the  method  selected  relies  upon 
considering  the  original  Decadal  Campaign  as  "truth";  if  every  instrument  is  flown,  100%  of 
the  benefit  is  realized. 

Hence,  the  instrument-measurement  scoring  is  useful  only  for  quantifying  relative 
relationships  amongst  instruments.  If  only  one  instrument  in  the  original  Decadal 
campaign  captures  a  particular  measurement,  then  by  default  100%  of  the  value  of  that 
measurement  is  traceable  to  that  instrument,  regardless  of  how  useful  that  particular 
instrument  actually  is.  If  multiple  instruments  in  the  original  Decadal  campaign  do  an 
equally  excellent  job  of  capturing  a  measurement,  all  of  them  must  be  flown  to  capture 
100%  of  the  benefit. 

Instrument-objective  satisfaction  was  calculated  by  first  converting  the  instrument- 
measurement  survey  scores  into  their  scaled  components  using  Table  9.  Then  the  scaled 
instrument-measurement  matrix,  1,  is  multiplied  by  the  normalized  measurement  matrix 
(Equation  8)  to  express  the  satisfaction  matrix,  f. 

/  =  /  * 

normalized 

Equation  8.  Satisfaction  Matrix 

The  satisfaction  matrix  expresses  how  well  each  instrument  satisfies  every  objective,  and  is 
then  normalized  by  objective  (Equation  9),  such  that  every  entry  in  the  normalized 
satisfaction  matrix  is  divided  by  the  sum  of  every  instrument’s  contribution  to  a  particular 
objective. 
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p  _ _ fu _ 

•^number  of  instruments  r 

Lk=l  Jl 

Equation  9.  Normalized  Satisfaction  Matrix 

The  weighted  satisfaction  matrix,  Wp,  can  then  be  found  (Equation  10),  which  relates  the 
absolutely  weighted  benefit  of  every  objective  to  the  normalized  satisfaction  matrix. 

Wp  W[,  absolute  ■*  ^ 

Equation  10.  Weighted  Satisfaction  Matrix 


3.1.5  Decadal  Campaign  Composition 

The  relationships  from  stakeholder  priorities  to  the  Decadal  Survey  Instruments 
have  been  mapped  in  a  series  of  three  matrices  which  compose  the  blue  and  purple  lower 
sections  of  the  CSTM  (Figure  16).  This  section  describes  the  population  of  the  three  orange 
sections.  Although  the  framework  is  designed  to  handle  multiple  architectures,  the  Decadal 
Survey  outlines  only  one. 

Instruments  in  the  Decadal  campaign  are  unique  to  one  of  the  proposed  missions, 
(although  a  particular  mission  may  include  several  instances  of  that  instrument).  The  39 
instruments  were  taken  from  the  mission  description  in  the  Decadal  Survey;  hence 
reassembling  the  relationships  was  trivial  (Table  10).  This  set  of  17  missions  formed  the 
basis  of  a  single  mission  set,  which  was  subsequently  mapped  to  three  tiers  rather  than 
specific  dates. 

The  single  architecture  of  the  Decadal  Survey  can  thus  be  described  in  one  table, 
rather  than  a  series  of  matrices.  These  relationships  will  be  revisited  in  Chapter  4  as 
variables  in  automated  campaign  architecting. 
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Tier  1:  2010-2013 

CLARREO 

thermal-IR 

spectrometer/interf 

erometer 

Near  IR-VIS-UV 

spectrometer/interf 

erometer 

GPS  receiver 

GPSRO 

advanced  GPS 

receiver 

SMAP 

L-band  radar 

L-band  radiometer 

ICESAT-II 

modified  GLAS 
(LIDAR) 

DESDynl 

L-band  inSAR 

IR  multi-beam  LIDAR 

Tier  2:  2013-2016 

XOVWM 

Ku  band  SAR 

scatterometer 

C-band  real-aperture 

scatterometer 

passive  (SRAD  with  K 
and  X  bands) 
radiometer 

HyspIRI 

thermal 

multispectral 

scanner 

spectrometer 

optical 

Hyperspectral 

imager 

ASCENDS 

1.57  or  2.06um 

LIDAR 

0.76  or  1.27  urn 

LIDAR 

IR  radiometer 

SWOT 

Ku-band  radar 

altimeter 

Ku-band  InSAR 

3-band  MW 

radiometer 

GEO-CAPE 

steerable  hi-res 

spectrometer 

NIR/VIS/UV  wide- 
area  spectrometer 

IR  correlation 

radiometer 

ACE 

cross-track  scanning 
cloud  radar 

multi-band  VIS/UV 
spectrometer 

Multi-beam  cross¬ 
track  dual¬ 
wavelength  LIDAR 

Multi-angle  multi¬ 
wavelength 
polarimeter 

Tier  3:  2016-2020 

LIST 

LIDAR  altimeter  with 
spatial  mapping 

PATH 

MW  spectrometer 

MW  radiometer 

GRACE-II 

Sat-to-Sat  ranger 
and  accelerometer 

SCLP 

dual  frequency  SAR 

passive  MW 
radiometer 

GACM 

SWIR/IR 

spectrometer 

MW  spectrometer 

UV/VIS 

spectrometer 

UV/VIS  differential 
absorption  LIDAR 

3D-WINDS 

non-coherent  wind 

lidar 

Coherent  wind  lidar 

Table  10.  Instrument  to  Mission  to  Mission  Set  to  Dates  Mapping 
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3.2  Comparison  to  the  Decadal  Survey 

The  relationships  in  the  CSTM  were  mapped  using  the  Decadal  Survey  and  NASA 
surveys  to  establish  truth.  One  mapping  that  the  Decadal  Survey  directly  enumerated  was 
that  between  the  17  missions  and  the  Decadal  objectives.  Using  the  CSTM,  a  similar 
summary  of  missions  to  objectives  was  calculated.  The  two  mapping  were  compared  to 
determine  how  effective  the  CSTM  is  in  replicating  the  Decadal  Survey. 

An  additional  survey  of  NASA  scientists  was  conducted  in  which  the  17  Decadal 
Survey  missions  were  evaluated  by  the  measurements  they  can  capture  (Appendix  A:  NASA 
Worksheet  instructions),  recombining  the  instruments  in  Table  10  for  easy  evaluation.  The 
mission-objective  satisfaction  calculations  were  calculated  using  the  instrument-objective 
equations  in  3.1.4. 

The  mission-objective  satisfaction  was  plotted  against  the  Decadal  Survey  (Figure 
23).  Although  the  Decadal  Survey  did  not  attempt  to  quantify  the  accrual  of  benefit,  it  did 
indicate  when  a  particular  mission  did  or  did  not  satisfy  an  objective,  allowing  the 
relationships  to  be  plotted  in  binary,  in  this  diagram,  the  CSTM  objectives,  sorted  by  panel, 
are  listed  on  the  x-axis  and  the  17  Decadal  Survey  missions  are  listed  on  the  y-axis.  The 
color  of  the  intersecting  square  indicates  the  relationship  explicitly  enumerated  in  the 
Decadal  Survey:  Black  squares  indicate  that  this  mission  does  contribute  to  this  objective; 
white  squares  indicate  that  this  mission  is  unrelated  to  this  objective.  The  CSTM  mission- 
objective  satisfaction  matrix  can  likewise  be  converted  to  binary  form  and  plotted  on  this 
chart.  The  number  in  the  intersecting  square  indicates  the  relationship  traced  through  the 
CSTM:  Ones  indicate  that  this  mission  does  contribute  to  this  objective,  zeros  indicate  that 
this  mission  is  unrelated  to  this  objective  (white  squares  with  no  numbering  are  zeros). 
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Solid  Earth 


Climate 


Weather 


Figure  23.  Binary  Mission-Objective  Satisfaction:  CSTM  versus  Decadal  Survey 

In  total,  the  Decadal  Survey  outlines  72  instances  of  mission-objective  satisfaction, 
whereas  the  CSTM  indicates  148,  indicating  that  the  CSTM  is  effective  in  locating 
unforeseen  synergies.  Every  "1"  located  in  a  white  square  indicates  benefit  that  the 
Decadal  Survey  did  not  anticipate.  Every  "0"  located  on  a  black  square  indicates  that  the 
CSTM  may  not  accurately  capture  all  the  necessary  relationships. 

Six  of  the  fourteen  relationship  shortcomings  occur  with  respect  to  the  Human 
Health  and  Security  panel's  objectives.  This  is  understandable  given  that,  "most  of  the 
missions  were  deemed  to  contribute  at  least  slightly  to  human  health  issues"  (National 
Research  Council,  2007,  p45);  the  exact  mission  contributions  to  Human  Health  were  not 
expressly  mapped  to  begin  with.  Similarly,  four  of  the  fourteen  shortcomings  are 
attribuatble  to  the  Water  panel,  particularly  the  "Snow  and  Cold  Land  processes".  Since 
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there  is  a  dedicated  Snow  and  Cold  Land  Processs  (SCLP)  mission,  it  is  unclear  what 
contributions  the  Decadal  Survey  expected  other  missions  to  make. 

The  remaining  shortcomings  are  attributable  to  four  missions.  The  NASA  Goddard 
Earth  Sciences  Exploration  Division  Chief  Engineer  was  interviewed  to  reason  through  the 
discrepancies: 

•  SMAP:  studying  surface  water,  while  a  logical  extension  of  studying  soil 
moisture  as  SMAP  intends,  is  unlikely. 

•  SWOT:  SWOT  is  intended  to  study  rivers  and  lakes,  and  is  tuned  to  making 
distinction  between  water  and  land;  hence  studying  ocean  circulation  is  not 
feasible. 

•  GEOCAPE:  the  lack  of  characterization  of  tropospheric  aerosols  potentially 
indicates  an  issue  with  the  mapping 

•  GACM:  it  is  unclear  how  the  Decadal  Survey  intended  to  use  GACM,  an 
atmospheric  composition  mission,  to  study  coastal  ecosystems.  However, 
the  lack  of  tropospheric  aerosol  characterization  potentially  indicates  an 
issue  with  the  mapping. 

An  analysis  of  the  individual  CSTM  elements  revealed  that  the  GACM  and  GEOCAPE 
instrument-measurement  characterizations  were  insufficient  to  capture  this  objective. 
This  was  identified  as  an  area  for  future  work. 

The  CSTM  is  sufficiently  capable  of  reproducing  the  Decadal  Survey  relationships. 
Although  a  few  discrepancies  were  noted  between  the  Decadal  and  CSTM  mappings,  the 
CSTM  identified  a  significant  number  unintended  benefits. 

3.3  Examination  of  Science  Traceability 

The  mapping  of  the  CSTM  was  compared  against  the  Decadal  Survey  to  establish 
validity  of  the  model.  The  intermediate  matrices  can  be  used  to  inform  campaign  design. 
The  value  of  science  fields,  instrument  types,  and  missions  can  be  analyzed.  The 
traceability  of  science  value  enables  a  cost-benefit  analysis. 
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First,  the  measurement  weighting  process  described  by  Equation  7  in  3.1.3  was 
utilized  to  weight  each  of  the  81  measurements  (Appendix  C:  Measurement  Weights).  The 
top  eleven  benefit  producing  measurements  are  displayed  in  Figure  24.  The  measurements 
depicted  in  the  chart  reflect  three  of  the  five  CEOS  science  areas  and  8  measurement 
categories. 


Top  Weighted  Specific  Measurements 


Figure  24.  Top  Eleven  Weighted  Measurements 

Decomposing  the  first  measurement  reveals  why  it  is  the  most  valuable.  The 
"Vegetation  State"  measurement  contributes  to  the  satisfaction  of  four  objectives: 

1.  Ecosystem  Function  (#1  objective) 

2.  Ecosystem  Structure  and  Biomass  (#2  objective) 

3.  Heat  Stress  and  Drought  (#24T  objective) 

4.  Vector-borne  and  Zoontic  Disease  (#24T  objective) 

The  value  of  this  measurement  is  logically  traceable  to  the  value  of  these  objectives: 
Measuring  vegetative  state  contributes  to  satisfying  ecosystem  function  and  ecosystem 
structure  objectives. 
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The  weighting  of  specific  measurements  can  be  summed  to  find  the  weightings  of 
the  28  measurement  categories  (Figure  25).  In  this  view  the  prevalence  of  certain  types  of 
measurements  is  much  clearer.  Aerosol  properties  are  the  dominant  category:  they  are 
required  by  nine  of  the  37  objectives.  This  plot  also  reveals  the  CEOS  categories  that  are 
not  relevant  to  the  Decadal  Survey:  Albedo  and  reflectance,  Ocean  Salinity,  and  Ocean  Wave 
height  and  spectrum. 


Decadal  Weighting  Applied  to  Measurement  Categories 


Figure  25.  Weighted  Measurement  Categories 


Finally,  the  weighted  measurements  can  be  reassembled  into  their  respective 
science  fields  (Figure  26).  This  plot  reveals  an  almost  linear  relationship  amongst  the 
science  areas:  atmospheric  science  is  extremely  important,  whereas  gravity  and  magnetic 
field  has  almost  no  value. 
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1]  Atmosphere  2]  Land  3]  Ocean  4]  Snow  and  Ice  5]  Gravity  and 

Magnetic  fields 


Figure  26.  Weighted  Science  Fields 


The  reason  for  this  vast  discrepancy  lies  in  the  urgency  of  stakeholder  needs. 
Applications  for  human  beings  tend  to  strongly  focus  on  immediate  concerns:  this  plot  can 
almost  be  redrawn  as  "urgency"  versus  "benefit".  This  pattern  is  a  well-know  effect  of 
earth  science,  as  indicated  in  this  NASA  plot  from  1989: 
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Figure  27.  Earth  Science  Timescale  (NASA,  1989) 
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A  badly  polluted  day  affects  a  person  much  sooner  than  subtle  shifts  in  the  geo¬ 
magnetic  field.  The  air  people  breathe,  the  spread  of  diseases,  the  hole  in  the  ozone  layer, 
and  the  weather  forecast  are  all  highly  dependent  on  atmospheric  science  and  are  all  short¬ 
term,  daily  responses.  Land  concerns,  such  as  forest  growth,  tend  to  on  more  of  a  seasonal 
cycle.  Ocean  applications  of  science,  such  as  the  conditions  of  fisheries,  tend  to  have  annual 
or  decadal  cycles.  Snow  and  ice  considerations  are  annual,  decadal,  and  centurial, 
particularly  the  advance  and  decline  of  glaciers.  Gravity  and  magnetic  issues  only  apply  on 
millennial  basis.  This  value-based  traceability  to  science  fields  strongly  indicates  the 
relationship  between  response  times  and  stakeholder  value. 

Similarly,  the  traceability  of  science  value  to  instruments  can  be  used  to  inform 
campaign  design.  The  weighting  of  instruments  described  in  3.1.4  can  be  used  to 
determine  both  the  measurements  and  objectives  not  actually  captured  in  the  Decadal 
campaign. 

1.8.1  stratospheric  ozone 

1.8.2  tropospheric  ozone 

1.8.3  ozone  precursors 

2.7.3  groundwater  storage 

3.2.1  surface  circuiation 

3.2.2  seafioor  topography 

3.2.3  coastai  upweiiing 

3.2.4  thermai  piumes 

3.2.5  river  piumes/sediment  fiuxes 

3.7.1  visibie  hydrospheric  poiiution  piumes 

5.1.2  magnetic  fieid  variations 


Table  11.  Measurements  not  Captured  by  Decadal  Instruments 

Many  of  the  measurements  seen  in 

Table  11  are  attributable  to  either  GEOCAPE  or  GACM,  which  were  ascertained  to  have 
insufficient  mappings.  However,  tracing  these  measurements  back  to  objectives  reveals 
that  no  Decadal  mission  actually  fulfills  the  "Ocean  Bathymetry"  objective,  which  depends 
only  on  ocean  surface  circulation  and  seafloor  topography  for  satisfaction,  in  the  Decadal 
Survey  this  objective  is  associated  with  the  SWOT  mission. 
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Because  of  this  disconnect  between  the  value  of  objectives  and  the  ability  of  Decadal 
missions  to  capture  this  value,  the  Decadal  campaign  will  not  accrue  100%  of  the  value  in 
the  system.  Since  no  Decadal  mission  individually  can  capture  the  relevant  measurements 
(assuming  no  synergies),  campaign  architecting  with  the  Decadal  set  of  instruments  will 
only  accrue  up  to  96.9%  of  the  available  benefit.  This  effect  is  seen  extensively  in  Chapters 
4  and  5,  which  discuss  the  accrual  of  benefit  over  time. 


The  Decadal  CSTM  also  enables  a  cost-benefit  analysis.  The  mission-objective 
satisfaction  calculations  used  to  produce  Figure  23  can  also  express  the  value  of  each 
mission.  In  Figure  28,  the  benefit  of  each  mission,  expressed  as  a  fraction  of  the  total 
benefit  in  the  campaign,  is  plotted  against  the  cost  of  each  mission,  as  listed  in  the  Decadal 
Survey.  It  is  desirable  to  fly  missions  that  contribute  high  amounts  of  benefit  but  are 
relatively  inexpensive:  hence  the  utopia  point  on  this  plot  is  the  upper-left  corner.  This 
analysis  suggests  that  the  best  value  missions  lie  along  the  line  roughly  drawn  between 
ACE  and  GPSRO,  including  SMAP,  HysplRl,  lCESat-11,  CLARREO,  LIST,  PATH,  GEO-CAPE,  and 
DESDynl.  This  also  suggests  that,  given  budgetary  constraints,  mission  like  GRACE-11, 
GACM,  XOVWM,  ASCENDS,  SCLP,  3D-Winds,  and  SWOT  should  be  removed  from  the 
campaign. 
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Figure  28.  Cost-Benefit  Plot  for  the  Decadal  Missions 
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Another  useful  visualization  of  the  mission-objective  satisfaction  matrix  is  to 
present  the  individual  panel's  contribution  to  the  value  of  each  mission  (Figure  29).  While 
essentially  conveying  the  same  information  as  Figure  28,  this  plot  displays  the  traceability 
of  value  to  different  panels.  ACE  is  by  far  the  most  beneficial  mission:  it  satisfies  all  six 
panels  to  at  least  some  degree  (the  only  mission  to  do  so),  and  makes  significant 
contributions  to  the  Weather,  Climate,  and  Land-Use  and  Ecosystems  panels.  This  makes 
sense  with  regards  to  the  Decadal  Survey  plan:  ACE  is  the  most  expensive  mission  and  has 
several  instruments  on-board,  the  most  prevalent  being  dedicated  to  Aerosols  and  Aerosol- 
Cloud  interactions.  This  also  fits  with  the  measurement  valuations  in  Figure  26,  which 
indicated  the  prevalence  of  atmospheric  science  in  conveying  value.  Conversely,  the  least 
valuable  mission,  SWOT,  only  satisfies  the  Solid  Earth  and  Water  panels,  and  not 
particularly  effectively. 


Panel  Benefit 


Figure  29.  Benefit  accrued  by  Panels  in  17  Original  Decadal  Missions 
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It  is  also  interesting  to  note  the  number  of  mission  for  which  a  panel  accrues  benefit 
(Table  12).  One  would  expect  the  Human  Health  and  Security  panel  to  gain  from  the  most 
number  of  missions:  it  did  not  recommend  a  dedicated  mission  because  it  sought  to  benefit 
from  the  other  panels'  data.  However,  it  only  accrues  benefit  in  12  of  the  17  missions, 
whereas  the  Water  panel  gains  from  14. 


Tab 


Number  of  Missions 
Contributing 

%  of  Campaign 

Water 

14 

82% 

Weather 

9 

53% 

Climate 

12 

71% 

Solid  Earth 

7 

41% 

Land-use 

8 

47% 

Human  Health 

12 

71% 

e  12.  Number  of  Missions  Contributing  to  Each  Panel 


3.4  Review  of  the  CSTM 


The  populated  CSTM  is  summarized  in  Figure  30.  The  Decadal  Survey  was 
decomposed  to  campaign  elements  which  were  then  related  through  CSTM  relationship 
matrices.  The  population  of  the  CSTM  yielded  the  following: 

•  Sutherland’s  stakeholder  priorities  were  used  to  weight  prioritized  panel 
objectives. 

•  A  common  set  of  measurements  was  derived  to  qualitatively  relate  objective 
satisfaction  to  the  instruments  and  missions  in  the  Decadal  Survey. 

•  The  CSTM  mapping  of  missions  to  objectives  was  compared  to  the  Decadal 
survey  mapping. 

•  The  traceability  of  science  value  to  measurements  was  analyzed  to  reveal 
the  science  field  value  to  stakeholders. 

•  Value  was  traced  to  instruments  and  missions,  enabling  a  cost-benefit 
analysis  of  the  Decadal  Survey  Missions. 


79 


Feasible 
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Architectures 


Solution  Space; 


•One  campaign 


Campaign 

Construction: 

•39  Instruments  mapped 
to  17  named  Missions 
•Missions  arranged  into 
three  tiers 


Mission  Sets 


38  Instruments 


From  NASA: 

•NASA  scientists  and  engineers  evaluated  the 
qualitative  relationship  between  Instruments 
and  Measurements 

•Potential  for  Synergistic  Relationships  to  be 
captured  by  mapping  Measurements  to  Measuremeni 


From  Decadal: 

•Explicit  statements  of  Objectives  taken  from 
each  Panel  in  DS 

•Objective  ranking  within  Panels  carried  from 
DS;  slope  of  relative  weighing  subjectively 
determined  from  DS 

•Objective  to  Measurement  relationships 
taken  from  Objective  descriptions  in  DS, 
compared  to  a  Common  Set  of  Measurements 
•Common  Set  of  Measurements  derived  from 
CEOS  database  (with  addition  of  all  specific  DS 
Measurements] 
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Figure  30.  Decadal  Survey  CSTM 

The  CSTM  reveals  a  great  deal  of  information  about  the  benefits  associated  with 
each  Decadal  mission.  At  this  point  only  a  static  view  of  the  campaign  value  has  been 
presented.  An  analysis  of  constraints  and  value  functions,  as  they  apply  to  time-dependent 
campaign  design,  will  be  presented  in  Chapter  4. 
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4  Campaign  Design  Using  the  CSTM 


The  campaign  design  methodology  described  in  this  chapter  enables  an  automated 
exploration  of  a  large  design  space  and  the  rapid  rearchitecting  of  a  campaign  given 
changing  assumptions,  as  described  in  Chapter  5.  Although  the  instruments  included  in  the 
Decadal  Survey  were  the  result  of  a  rigorous  selection  process,  the  assignment  of 
instruments  to  missions  and  the  assignment  of  missions  to  dates  were  conducted  as  round¬ 
table  discussions  amongst  panelists,  rather  than  being  approached  as  an  optimizable 
problem.  This  chapter  reviews  the  guidance  provided  by  the  Decadal  Survey  with  regards 
to  campaign  design.  It  examines  the  constraints  that  limit  these  assignment  processes.  It 
provides  a  set  of  metrics  for  evaluating  alternate  campaign  conceptions.  Finally,  it  presents 
an  algorithm  for  the  assignment  of  missions  to  dates.  Chapter  5  will  discuss  the  application 
of  these  constraints,  value-functions,  and  algorithm  to  campaign  design  outside  of  the 
Decadal  Survey  assumptions.  The  objectives  of  this  chapter  are: 

Objectives  of  Campaign  Design  Using  the  CSTM 

•  To  replicate  the  decision  logic  of  the  Decadal  Survey  by  implementing  constraint 
and  value  functions  using  algorithm  experimentation 

•  To  validate  the  methods  and  model  introduced  in  this  thesis  by  comparing  results 
using  the  Decadal  Survey  as  “truth'' 

Many  of  the  constraints  and  value  functions  discussed  in  this  chapter  were 
originally  proposed  in  Colson’s  Master’s  thesis  (Colson,  2008).  This  chapter  presents  a 
refinement  of  many  of  these  ideas,  in  addition  to  their  application  to  the  CSTM.  This 
chapter  is  organized  into  the  following  sections: 

•  Section  4.1:  Campaign  Constraints.  This  section  discusses  the  classes  of 
constraints  that  apply  to  campaign  design. 

•  Section  4.2:  Campaign  Value  Functions.  This  section  describes  the  rationale  and 
calculation  of  campaign  value-functions. 

•  Section  4.3:  Scheduling  Algorithms.  This  section  discusses  the  development  of 
an  algorithm  for  the  automated  campaign  scheduling. 

•  Section  4.4:  Summary 
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4.1  Campaign  Constraints 


Rearchitecting  of  the  Decadal  campaign  requires  manipulating  the  relationships 
between  the  higher  order  campaign  elements — those  represented  in  the  orange  "campaign 
construction"  block  of  the  CSTM  (Figure  30  in  Section  3.4).  These  relationships  are  limited 
by  different  classes  of  constraints  applied  on  different  levels.  This  section  describes  the 
Decadal  Survey  guidance  regarding  design  constraints.  Then,  Colson's  constraints  are 
modified  to  apply  to  the  CSTM.  Finally,  other  classes  of  constraints  are  considered.  This 
section  concludes  with  a  discussion  of  feasibility  and  a  summary  of  the  application  of  these 
constraints. 

4.1.1  Constraint  Guidance  from  the  Decadal  Survey 

The  Decadal  Survey  outlines  the  decision  processes  utilized  by  the  panels  to 
prioritize  mission  concepts  (Figure  31).  Although  each  panel  underwent  a  unique  process 
to  arrive  at  their  final  set  of  proposed  mission  concepts,  these  guidelines  can  inform  the 
application  of  constraints. 


BOX  ES.1  CRITERIA  USED  BY  THE  PANELS  TO  CREATE  RELATIVE  RANKINGS  OF  MISSIONS 


•  Contribution  to  the  most  important  scientific  questions  facing  Earth  sciences  today  (scientific  merit, 
discovery,  exploration) 

•  Contribution  to  applications  and  policy  making  (societal  benefits) 

•  Contribution  to  long-term  observational  record  of  Earth 

•  Ability  to  complement  other  observational  systems,  including  planned  national  and  international 
systems 

•  Affordability  (cost  considerations,  either  total  costs  for  mission  or  costs  per  year) 

•  Degree  of  readiness  (technical,  resources,  people) 

•  Risk  mitigation  and  strategic  redundancy  (backup  of  other  critical  systems) 

•  Significant  contribution  to  more  than  one  thematic  application  or  scientific  discipline 

Note  that  these  guidelines  are  not  in  priority  order,  and  they  may  not  reflect  all  of  the  criteria  considered 
by  the  panels. 


Figure  31.  Mission  Prioritization  Guidance  (National  Research  Council,  2007) 
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Each  of  these  eight  criterion  reflect  possible  constraints.  The  first  two  points 
highlight  the  balancing  act  between  discovery  and  application  bias  of  research  missions 
that  the  Decadal  Survey  underwent.  Missions  must  address  societal  applications  or 
research  potential  future  applications  by  answering  important  science  questions. 

Although  the  CSTM  assumes  a  uniform  level  of  mission  specificity  (as  described  in  2.1.4)  if 
the  objectives  of  the  Decadal  campaign  are  also  to  be  architected,  objectives  must  be 
similarly  balanced.  The  third  point  indicates  the  importance  of  data  continuity,  a 
constraint  identified  by  Colson.  The  fourth  point  highlights  the  desirability  of  synergistic 
effects.  Although  mission  independence  was  assumed,  it  is  possible  apply  synergy  as  a 
constraint.  The  fifth  point  stresses  the  importance  of  cost  and  budget  in  mission  selection, 
another  constraint  identified  by  Colson.  The  sixth  criterion  identifies  the  limitations 
imposed  by  technological  readiness.  TRL  can  express  either  a  probabilistic  risk  valuation 
or  can  be  considered  a  strict  limit  for  mission  scheduling,  as  proposed  by  Colson.  The 
seventh  point  outlines  the  need  for  strategies  for  campaign  element  failures.  This  does  not 
suggest  a  particular  constraint  per  se,  but  does  highlight  the  need  for  rapid  contingency 
campaign  architecting.  The  final  point  expresses  the  importance  of  mission  breadth;  the 
Decadal  committee  attempted  to  ensure  that  missions  represented  diverse  interests.  The 
outcome  of  this  effort  is  seen  in  the  traceability  of  value  to  measurements  (Figure  26),  and 
the  number  of  panels  satisfied  by  each  mission  (Table  12). 
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4.1.2  Colson's  Constraints  Applied  to  the  Decadal  Survey  CSTM 

Colson  adopted  four  of  the  eight  criteria  outlined  by  the  Decadal  Survey  for  mission 
prioritization  (Figure  32).  The  following  sections  outline  how  they  are  applied  to  the 
CSTM.  Although  the  specific  implementations  do  differ,  the  concepts  are  reflective  of  the 
Decadal  decision  logic. 


Decision  Rule  1:  Campaign  Budget 

Missions  within  a  campaign  were  scheduled  such  that  the  expenditure  rate,  carefully  based  on 
mission  costs  shown  in  Table  2.1,  did  not  exceed  the  prescribed  budget  (baseline  budget  of  $750 
million  per  year). 


Decision  Rule  2:  Technology  Readiness  Level 

Missions  were  scheduled  so  that  no  flights  were  cued  before  their  technology  readiness  date.  In 
the  baseline  OPN  Scheduler,  these  dates  were  taken  from  the  Decadal  Survey,  as  shown  in 
Table  2.1. 


Decision  Rule  3:  Data  Continuity 

The  OPN  Scheduler  forced  mission  overlap  and  continuous  measurements  in  accordance  with 
the  recommendations  presented  in  the  Decadal  Survey  (baseline  OPN  Scheduler  case).  Flights 
were  ordered/scheduled  to  guarantee  any  required  overlap  in  data  coverage. 

Rule  3a;  Cumulative  Measurements 

Certain  missions  were  forced  to  overlap  in  time,  when  the  cumulative  measurements  were 
required. 

Rule  3b:  Measurement  Developments  and  Technology  Roadmaps 

The  scheduler  was  designed  such  that  a  specific  ordering  of  similar  subsets  of  missions  was 
maintained,  whenever  these  measurements  were  part  of  a  long-term  measurement  development 
plan  or  technology  roadmap  for  other  flights. 

Rule  3c:  Latest  Dates 

Latest  possible  launch  dates  were  implemented  in  specific  flights  to  ensure  they  happen  before 
their  latest  recommended  execution  in  the  Decadal  Survey. 


Decision  Rule  4:  Value  Delivery  Fairness 

In  the  baseline  case,  the  scheduler  was  only  allowed  to  choose  missions  where  one  of  the  top 
two  highest  value  delivery  objectives  delivered  value  to  satisfy  one  of  the  two  science 
communities  with  the  largest  “uncaptured  benefit.” 


Figure  32.  Colson's  Four  Constraints  (Colson,  2008) 
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4.1.2. 1  Mission  Costs  and  Annuai  Budget 

The  first  constraint  Colson  imposed  on  campaign  design  is  an  annual  spending  limit 
which  a  campaign  expenditure  rate  cannot  exceed.  This  limitation  constrained  the 
frequency  of  mission  launch.  No  translation  was  necessary  to  apply  this  to  the  CSTM; 
however,  Colson’s  rationale  will  be  explained. 

NASA's  budget  for  earth  science  and  the  Decadal  Survey  campaign  is  limited  in  size 
and  scope.  While  funding  may  sporadically  appear  over  the  course  of  a  year,  the  general 
trend  reflects  a  relatively  stable  program  budget  (Figure  4).  It  is  assumed  that  the  budget 
will  remain  at  a  constant  level  for  the  duration  of  a  campaign  and  that  the  annual  funding 
profile  will  average  out  linearly. 

It  is  also  assumed  that  every  mission  will  accrue  costs  over  time  according  to  a 
predictable  distribution  (Larson  &  Wertz,  1999  p  804).  A  standard  distribution,  for 
example,  can  represent  the  spending  profile  of  a  mission  over  time:  during  early  studies 
little  money  is  actually  spent;  as  the  design  matures  staffing  increases  and  hardware  is 
purchased;  as  the  assembly  begins  the  design  staff  moves  on  to  other  projects,  and  begins 
to  decline;  after  launch,  only  a  small  operations  cost  remains. 

The  cumulative  spending  profile  of  multiple  missions  over  time  represents 
campaign  spending  (Figure  33).  Because  a  campaign  is  limited  to  a  linear  annual  budget 
(or  spending  limit),  the  most  efficient  scheduling  will  overlap  mission  spending 
distributions  such  that  the  combined  mission  spending  is  closest  to  this  limit.  Although 
each  mission  individually  has  a  normal  spending  distribution,  the  campaign  can  be 
expressed  as  a  sequence  of  step  functions,  with  only  one  mission  being  developed  at  a  time 
(Figure  34). 


Figure  33.  Campaign  Spending  with  Standard  Distributions  (Colson,  2008) 


85 


Figure  34.  Campaign  Spending  with  Step  Distribution  (Colson,  2008) 


Using  the  assumption  of  annual  budget  linearity  and  the  assumption  of  step- 
function  mission  costing  enables  the  conversion  of  costs  to  time  (Equation  11).  A  sequence 
of  missions  can  be  scheduled  at  a  rate  where  time  between  mission  launches  is  a  function 
of  annual  budget  and  mission  costs. 


t  = 


Mission  Cost  ($M) 

Annual  Budget  (  ^ 

^  \year) 


Where  t  is  time  elapsed  between  mission  launch  dates 


Equation  11.  Time  as  a  Function  of  Budget 


The  budget  constraint  dictates  that  annual  spending  cannot  exceed  the  annual 
budget.  This  constraint  applies  to  the  mission  level  of  the  CSTM;  although  cost  is  driven  by 
instruments,  is  a  primary  attribute  of  missions. 

Mission  Cost  and  Annual  Budget  Constraint: 

•  Annual  spending  cannot  exceed  the  annual  budget 

4. 1.2. 2  Technology  Readiness  Level 

The  second  constraint  Colson  applied  to  campaign  design  is  a  Technology  Readiness 
Level  (TRL)  date.  Missions  cannot  be  flown  before  the  instrument  onboard  have  actually 
been  developed  and  tested.  Colson  assumed  that  the  launch  dates  attributed  to  missions  in 
the  Decadal  Survey  were  representative  of  the  earliest  dates  a  mission  would  be 
technologically  ready. 
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This  constraint  is  implemented  on  the  instrument  level  of  the  CSTM  rather  than  the 
mission  level.  Although  for  some  missions  engineering  challenges  delay  the  expected 
readiness  date  (such  as  SMAP  which  requires  a  rotating  6m  antenna),  it  is  assumed  that 
this  limitation  is  a  quality  of  instruments.  Additionally,  it  is  assumed  that  the  TRL  date  of  a 
mission  is  equal  to  the  latest  TRL  of  the  instruments  onboard. 

Technology  Readiness  Level  Constraint: 

•  Missions  cannot  be  flown  before  the  instruments  on-board  have  all  reached 
technological  maturity 


4. 1.2.3  Data  Continuity 

The  third  constraint  Colson  applied  to  campaign  design  is  a  need  for  data  continuity. 
However,  for  the  CSTM  it  was  determined  that  making  data  continuity  a  constraint  was 
unfeasible.  This  section  describes  the  rationale  for  making  data  continuity  a  value  function. 

Colson  include  three  sub-definitions  of  the  data  continuity  constraint:  missions  with 
known  synergies  must  be  in-orbit  during  each  other's  lifetimes,  missions  that  contained 
early  versions  of  later  tier  instruments  must  fly  before  their  later  tier  counterparts,  and 
missions  replacing  current  assets  must  be  in-orbit  before  those  assets  reach  end-of-life. 

For  Colson's  simulation  these  were  easily  enforceable  limitations,  as  they  only  affected  a 
small  number  of  missions. 

While  the  Decadal  Survey  expressed  contributions  to  long-term  observational 
records  as  an  important  factor  in  decision  making,  it  did  not  have  a  systematic  view  for 
considering  measurements  over  time.  As  part  of  NASA's  emphasis  on  climate  science,  a  list 
of  28  Essential  Climate  Variables  has  been  developed  by  NASA  Goddard,  and  corresponding 
mission-measurement  profiles  have  been  assembled  (NASA,  2009).  There  is,  however,  no 
distinction  made  as  to  the  relative  importance  of  one  measurement  over  another.  These 
ECV's  were  translated  to  the  common  set  of  measurements  to  highlight  34  measurements 
desiring  data  continuity  (Table  13). 
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34  Measurements  needing  continuity 

1.1.1  aerosol  height/optical  depth 

1.10.11  C02  concentrations 

1.1.2  aerosol  composition,  physical  and  chemical 
properties 

2.1.1  Albedo  and  reflectance 

1.1.3  aerosol  scattering  properties 

2.3.2  soil  moisture 

1.1.4  aerosol  extinction  profiles 

2.4.2  vegetation  state 

1.1.5  aerosol  size  and  size  distribution 

2.4.4  canopy  density 

1.2.1  Atmospheric  temperature  fields 

2.6.2  landcover  status 

1.3.1  Water  vapor  profiles 

2.7.1  river  and  lake  elevation 

1.5.1  cloud  top  temperature 

3.1.1  Ocean  color 

1.5.2  Cloud  type 

3.2.1  surface  circulation 

1.5.3  Cloud  amount/distribution 

3.3.1  Ocean  salinity 

1.6.1  cloud  height/optical  thickness 

3.4.1  Ocean  surface  wind  speed 

1.6.2  cloud  particle  size  distribution 

3.4.2  Ocean  surface  wind  direction 

1.7.1  Precipitation  rate 

3.5.1  Surface  temperature  (ocean) 

1.8.1  stratospheric  ozone 

4.1.1  ice  sheet  volume 

1.8.2  tropospheric  ozone 

4.1.2  Glacier  surface  elevation 

1.9.2  spectrum  of  earth  IR  radiance 

4.2.4  snow  cover 

1.10.4  Benchmark  tracer  data  (C02,  CO,  HD0/H20, 
NOy,  N20,  CH4,  halogen  source  molecules) 

4.3.1  Sea  ice  thickness 

Table  13.  Measurements  Desiring  Data  Continuity 


While  continuity  of  measurements  and  instrument  overlap  is  desirable,  the  CSTM 
does  not  treat  it  as  a  constraint.  The  Decadal  Survey  made  note  of  continuity 
considerations,  but  did  not  require  them.  Instead,  for  the  CSTM  the  number  of  breaks  in 
continuous  measurement  is  evaluated  for  each  campaign  architecture  as  a  secondary  value 
function. 


Data  Continuity  Value-Function: 

•  Assuming  a  notional  average  mission  life,  count  the  number  of  breaks  incontinuous 
coverage  of34  key  measurements. 


4. 1.2.4  Fairness 

The  final  constraint  Colson  applied  to  campaign  design  is  a  conception  of  fairness. 
Although  the  objectives  in  the  Decadal  campaign  represented  the  interests  of  a  diverse 
science  community,  Colson  believed  it  was  necessary  to  constrain  the  scheduling  of 
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missions  in  a  manner  that  distributed  the  accrual  of  benefit  over  time  to  different  science 
panels.  Colson  evaluated  fairness  with  respect  to  either  of  the  two  least  satisfied  panels. 

The  CSTM  assumes  that  the  Decadal  Survey  has  already  allocated  all  of  the  possible 
mission  value  in  the  system.  Hence,  fairness  is  a  constraint  that  affects  the  ordering  of 
missions  based  upon  the  amount  of  uncaptured  benefit  in  the  system.  The  traceability  of 
value  in  the  CSTM  presented  several  opportunities  for  an  algorithmic  fairness  routine. 
Experimentation,  as  will  be  described  in  4. 3. 3. 2,  reveal  an  "impartial"  definition  of  fairness, 
applied  on  the  panel  level,  to  most  accurately  reproduce  the  Decadal  Survey  decision  logic. 


The  fairness  constraint  requires  an  equal  weighing  of  all  panels,  although  the 
satisfaction  of  those  panels  is  still  subject  to  the  other  CSTM  relationship  weightings. 
Fairness  requires  minimizing  the  deviation  between  the  benefit  accrued  over  time  by 
different  panels  (Figure  35) 


Figure  35.  Fairness  Implementation 


In  this  example  the  deviation  of  the  least  satisfied  option  from  some  notional 
reference  is  marked  in  black.  The  application  of  the  fairness  constraint  requires  the  next 
mission  selected  to  attempt  to  close  the  gap  in  some  way. 

Fairness  Constraint: 

•  Minimize  the  deviation  of  the  relative  benefit  accrued  by  each  panel  over  time 
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4.1.3  Other  Constraints 

Another  constrain  implemented  in  the  CSTM  is  the  concept  of  mission  scope. 
Missions  should  not  be  assembled  with  an  unreasonably  large  number  of  instruments.  The 
Decadal  Survey  recommended  a  balance  between  mission  sizes.  Although  they  categorized 
by  mission  cost,  their  campaign  of  17  missions  included  a  blend  of  single  and  multiple 
instrument  missions  (Table  10).  Campaign  design  with  the  CSTM  should  limit  the 
assignment  of  instruments  to  missions. 

Scope  Constraint 

•  Limit  number  of  instruments  per  mission 

Additionally,  because  the  Decadal  Survey  is  a  research  campaign,  the  constraint  that 
each  mission  can  be  flown  only  once  is  added.  In  reality,  as  the  Decadal  mission 
demonstrate  their  utility  in  providing  applications  they  will  be  operationalized,  and  flying 
multiple  copies  of  the  same  mission  will  be  considered. 

Operationalizing  Constraint 

•  Each  mission  can  only  be  scheduled  once 


4.1.4  Summary  of  Constraints 

The  Decadal  Survey  implicitly  recommends  a  set  of  campaign  design  constraints. 
Colson  codified  these  into  four  classes:  cost,  TRL,  data  continuity,  and  fairness.  Constraints 
of  scope  and  operationalizing  were  added  based  upon  Decadal  Survey  recommendations. 
The  implementation  of  these  constraints  in  the  CSTM  is  summarized  in  the  following  figure: 
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Figure  36.  Constrained  CSTM 

Campaign  design  relies  upon  recreating  the  mappings  in  the  "Solver"  block  of  Figure 
36.  Constraints  are  applied  to  the  CSTM  to  ensure  architectural  feasibility  of  campaign 
design.  Value  functions,  such  as  data  continuity,  are  used  to  compare  viable  campaign 
architectures  against  each  other.  Section  4.2  introduces  the  primary  value  functions  used 
to  evaluate  CSTM  campaigns. 

4.2  Valuing  a  Campaign 

While  constraints  are  used  to  limit  campaign  design  options  to  feasible  solutions, 
value  functions  are  used  to  compare  designs.  Colson  assumed  a  single  value  function,  time 
discounting,  for  campaign  architecture  differentiation,  in  addition  to  his  method  and  the 
data  continuity  function  discussed  in  Section  4. 1.2.3,  two  other  metrics  are  applicable  to 
the  CSTM:  the  percentage  of  total  value  accrued  in  the  system  and  the  synergistic  benefits 
of  scheduling. 
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The  percentage  of  total  value  accrued  expresses  the  sum  of  all  objective  satisfaction 
completed  in  a  campaign  relative  to  the  total  possible  value  of  the  Decadal  campaign,  if  a 
campaign  is  budget  or  time  constrained,  it  may  not  be  possible  to  fly  every  mission;  hence, 
a  particular  campaign  architecture  may  not  satisfy  every  Decadal  objective.  Campaign 
architectures  dominated  by  high  benefit  per  cost  missions  will  accomplish  more  given 
these  constraints.  As  was  discussed  in  Section  3.3,  the  campaign  proposed  in  the  Decadal 
Survey  fails  to  accomplish  every  objective — missing  out  on  the  Ocean  Bathymetry  objective 
and  the  corresponding  percentage  of  total  value  associated  with  that  objective. 

Second,  if  the  synergistic  effects  of  overlapping  measurements  are  quantified,  the 
additional  benefit  provided  through  concurrent  scheduling  of  instruments  can  be  captured. 
This  metric,  like  quantifying  measurement  synergies,  is  identified  as  an  area  for  future 
work. 

Colson's  time  discounting  reflects  the  concept  of  the  time  value  of  money.  The 
principle  states  that,  without  considering  inflation,  a  dollar  today  is  worth  more  than  a 
dollar  in  a  year,  as  today’s  dollar  can  be  invested  and  earn  a  return  for  the  year.  Colson 
applied  the  principle  of  net  present  value  to  the  accrual  of  benefit  over  time  through  the 
use  of  value  discounting  (Equation  12).  The  value  in  the  future  at  time  t  is  equal  to  the 
starting  value  modified  by  the  discount  rate. 

Valued)  = 

(!+«;)' 

Equation  12.  Present  Value  Discounting 

Every  permutation  in  the  campaign  solution  space  will  have  a  unique  arrangement 
of  instruments  and  missions,  it  is  assumed  that  benefit  is  accrued  at  the  time  of  mission 
launch;  hence  the  value  delivered  by  each  instrument  can  be  discounted  by  when  that 
instrument  launches.  Colson  assumed  the  following  discount  rates  for  the  objectives  of 
different  panels: 
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Science  Panel 

Depreciation  Rate  (%) 

Climate 

15 

Water  Resources 

10 

Health  and  Human  Security 

10 

Weather 

10 

Ecosystems 

10 

Solid  Earth 

5 

Table  14.  Panel  Discount  Rates  (Colson,  2008) 

A  discount  rate  of  10%  is  a  standard  assumption  for  analysis,  and  was  consequently 
assigned  to  the  benefit  of  most  science  panels  (Larson  &  Wertz,  1999,  p  807).  Climate  was 
given  a  higher  discount  rate,  reflective  of  the  apparent  urgency  of  understanding 
anthropomorphic  climate  change.  Solid  Earth  objectives  were  given  a  lower  rate,  reflective 
of  the  priorities  seen  in  Section  3.3. 

Colson  assumed  that  every  campaign  architecture  will  include  all  17  original 
Decadal  missions — discounted  benefit  was  his  only  metric  to  separate  mission  sequences. 
With  the  CSTM,  it  is  necessary  to  consider  architectures  where  different  instruments  are 
never  flown.  Hence,  the  use  of  Colson’s  metric  is  applied  on  top  of  the  percentage  of 
Decadal  value  accrued. 

Campaign  architecting  with  the  CSTM  relies  upon  Colson’s  discounted  value  metric 
as  applied  of  the  percentage  of  total  value  accrued.  Data  continuity  is  calculated,  but  serves 
as  a  secondary  consideration.  Quantifying  synergistic  effects  for  use  as  a  value-function  is 
an  area  for  future  work. 


Value  Functions 

•  Primary:  The  present  value  of  a  mission  is  calculated  by  discounting  the  weighted 
satisfaction  of  that  mission  with  respect  to  the  different  panels.  The  discounted 
value  of  every  mission  in  a  campaign  is  summed  and  expressed  as  a  percentage  of 
total  possible  value. 

•  Secondary:  The  number  ofData-continuity  gaps  is  counted  as  a  secondary  metric 

•  Tertiary:  Studying  synergistic  effects  is  identified  as  a  possible  metric  requiring 
future  work 
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4.3  An  Algorithm  for  Replicating  the  Decadal  Survey  Decision  Logic 

The  previous  two  sections  discussed  the  constraints  and  value  functions  applicable 
to  campaign  architecting  with  the  CSTM.  This  section  discusses  the  application  of  the 
Decadal  Survey  decision  logic  to  an  algorithm  for  campaign  scheduling.  First,  the  guidance 
provided  by  the  Decadal  Survey  is  analyzed.  Then,  a  reference  schedule  is  defined.  Third, 
the  development  and  structure  of  an  enhanced  algorithm  is  discussed.  This  section 
concludes  with  a  summary. 

4.3.1  Algorithm  Guidance  from  the  Decadal  Survey 

The  Decadal  Survey  enumerated  a  set  of  programmatic  decision  strategies  and  rules 
that  can  be  used  to  inform  campaign  development  (Figure  37).  This  list  includes  three 
primary  principles:  leverage  international  efforts,  manage  technology  risk,  and  respond  to 
budget  pressures  and  shortfalls. 

The  rationale  for  these  principles  is  summarized: 

1.  Leverage  international  efforts:  earth  science,  by  definition,  applies  to  a  greater 
community  than  the  United  States.  The  benefits  of  an  earth  science  campaign 
are  not  exclusive,  and  the  costs  do  not  need  to  be.  Taking  advantage  of  other 
space  program’s  missions  will  help  ensure  a  robust  campaign. 

2.  Manage  technology  risk:  technological  development  can  be  a  huge  risk,  not  only 
in  increases  to  the  costs  of  a  particular  mission,  but  in  the  progression  of  an 
entire  campaign.  A  campaign  can  avoid  technology  issues  by  enacting  individual 
mission  development  campaigns. 

3.  Respond  to  budget  pressures  and  shortfalls:  cost  and  budget  concerns  affect  the 
entire  campaign,  and  changes  need  to  be  evaluated  with  respect  to  the  whole 
program.  Large  cost  overruns  on  one  mission  can  put  the  remainder  of 
campaign  at  risk.  However,  if  a  mission  is  at  risk  of  being  cancelled,  it  is  best  to 
degrade  its  performance  parameters,  and  therefore  cost,  as  much  as  possible  to 
keep  the  mission  in  the  campaign.  Even  if  a  particular  mission  is  cancelled,  the 
objectives  it  would  have  satisfied  should  not  be  ignored. 
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Included  in  this  list  is  the  specific  recommendation  that  a  campaign  should 
"sequence  missions  according  to  technological  readiness  and  budget-risk  factors"  (Figure 
37).  The  Decadal  Survey  acknowledges  that  this  principle  biases  a  campaign  schedule 
towards  a  "cheaper  first"  approach.  However,  considering  the  factors  that  put  campaign  at 
risk,  the  Decadal  determined  that  this  was  the  best  solution.  This  recommendation  is  the 
basis  for  any  discussion  into  scheduling  algorithms. 


BOX  ES.2  PROGRAMMATIC  DECISION  STRATEGIES  AND  RULES 


Leverage  International  Efforts 

•  Restructure  or  defer  missions  if  international  partners  select  missions  that  meet  most  of  the  measure¬ 
ment  objectives  of  the  recommended  missions;  then  (1)  through  dialogue  establish  data-access  agreements, 
and  (2)  establish  science  teams  to  use  the  data  in  support  of  the  science  and  societal  objectives. 

•  Where  appropriate,  offer  cost-effective  additions  to  international  missions  that  help  extend  the  values 
of  those  missions.These  actions  should  yield  significant  information  in  the  identified  areas  at  substantially  less 
cost  to  the  partners. 

Manage  Technology  Risk 

•  Sequence  missions  according  to  technological  readiness  and  budget  risk  factors. The  budget  risk  consid¬ 
eration  may  favor  initiating  lower-costmissionsfirst.  However,  technology  investments  should  be  made  across 
all  recommended  missions. 

•  Reduce  cost  risk  on  recommended  missions  by  investing  early  in  the  technological  challenges  of  the 
missions.  If  there  are  insufficient  funds  to  execute  the  missions  in  the  recommended  time  frames,  it  is  still 
important  to  make  advances  on  the  key  technological  hurdles. 

•  Establish  technology  readiness  through  documented  technology  demonstrations  before  a  mission's 
development  phase,  and  certainly  before  mission  confirmation. 

Respond  to  Budget  Pressures  and  Shortfalls 

•  Delay  downstream  missions  in  the  event  of  small  (~10  percent)  cost  growth  in  mission  development. 
Protect  the  overarching  observational  program  by  canceling  missions  that  substantially  overrun. 

•  Implement  a  system-wide  independent  review  process  that  permits  decisions  regarding  technical 
capabilities,  cost,  and  schedule  to  be  made  in  the  context  of  the  overarching  science  objectives.  Programmatic 
decisions  on  potential  delays  or  reductions  in  the  capabilities  of  a  particular  mission  could  then  be  evaluated 
in  light  of  the  overall  mission  set  and  integrated  requirements. 

•  Maintain  a  broad  research  program  under  significantly  reduced  agency  funds  by  accepting  greater 
mission  risk  rather  than  descoping  missions  and  science  requirements.  Aggressively  seek  international  and 
commercial  partners  to  share  mission  costs.  If  necessary,  eliminate  specific  missions  related  to  a  theme  rather 
than  whole  themes. 

•  In  the  event  of  large  budget  shortfalls,  re-evaluate  the  entire  set  of  missions  in  light  of  an  assessment  of 
the  current  state  of  international  global  Earth  observations,  plans,  needs,  and  opportunities.  Seek  advice  from 
the  broad  community  of  Earth  scientists  and  users  and  modify  the  long-term  strategy  (rather  than  dealing 
with  one  mission  at  a  time).  Maintain  narrow,  focused  operational  and  sustained  research  programs  rather 
than  attempting  to  expand  capabilities  by  accepting  greater  risk.  Limit  thematic  scope  and  confine  instrument 
capabilities  to  those  well  demonstrated  by  previous  research  instmments. 


Figure  37.  Programmatic  Decision  Guidance  (National  Research  Council,  2007) 
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4.3.2  Establishing  a  Reference 

An  algorithm  for  campaign  scheduling  needs  to  be  validated  against  a  reference. 

The  Decadal  Survey  proposes  both  a  reference  schedule  and,  as  discussed  in  the  previous 
section,  a  decision  rule  for  scheduling.  This  section  describes  the  establishment  of  a 
reference  using  these  two  sources  in  addition  to  some  of  the  constraints  discussed  in 
section  4.1.  Additionally,  this  section  introduces  the  plotting  of  benefit  over  time  as  a  tool 
for  informing  campaign  development. 

4. 3.2. 1  The  Reference  Schedule 

The  Decadal  Survey  stops  short  of  recommending  a  specific  timeline  for  the 
development  and  launch  of  its  17  missions,  it  does,  however,  imply  a  preferred  order  that 
has  been  arranged  in  accordance  with  its  proposed  algorithm  (Appendix  B:  Reference 
Sequence).  This  sequence  is  referred  to  as  the  "Reference"  case  and  is  summarized  in  Table 
15  below: 


Decadal  Survey  Reference  Case 


Tier 

Mission 

Readiness 

Date 

DS 

FY06 

Cost 

(M$) 

Tier  1 

2010-2013 

CLARREO 

2010 

265 

GPSRO 

2012 

150 

SMAP 

2012 

300 

ICESat-II 

2010 

300 

DESDynl 

2010 

700 

Tier  2 

2013-2016 

XOVWM 

2013 

350 

HyspIRI 

2015 

300 

ASCENDS 

2013 

400 

SWOT 

2013 

450 

GEO-CAPE 

2015 

550 

ACE 

2015 

800 

Tier  3 

2016-2020 

LIST 

2017 

300 

PATH 

2015 

450 

GRACE-II 

2016 

450 

SCLP 

2016 

500 

GACM 

2017 

600 

3D-Winds 

2016 

650 

Table  15.  Reference  Sequence 
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Each  mission  in  the  Decadal  Survey  was  assigned  a  readiness  date  based  upon  a 
combination  of  fairness,  TRL,  and  data  continuity  considerations.  As  these  considerations 
are  not  explained,  it  is  assumed  this  date  is  analogous  to  the  TRL  date  metric  utilized  by 
Colson.  Each  mission  is  allocated  to  the  tier  which  contains  its  readiness  date,  with  the 
exception  of  PATH,  which  is  confusingly  attributed  a  date  of  "about  2010-2015"  (National 
Research  Council,  2007,  p  125).  This  architecture  thus  reflects  the  decision  rule 
recommendation  to  sequence  missions  first  by  technology  readiness  risk  by  putting 
missions  into  three  TRL  tiers. 

The  first  tier  includes  the  missions  CLARREO,  GPSRO,  SMAP,  lCESat-11,  and  DESDynl. 
Reflecting  the  second  algorithmic  decision  principle,  to  prioritize  missions  by  budget  risk, 
these  five  missions  are  arranged  by  increasing  cost,  with  the  exception  of  GPSRO.  This 
exception  explainable  by  a  data  continuity  consideration,  as  GPSRO  is  designed  to  replace 
the  COSMIC  mission  (which  is  expected  to  last  until  2012),  also  measuring  occultation. 

The  second  tier  includes  XOVWM,  HysplRl,  ASCENDS,  SWOT,  GEO-CAPE,  and  ACE. 
These  missions  are  similarly  prioritized  by  cost  within  this  tier,  with  the  exception  of 
HysplRl.  The  Decadal  Survey  provides  no  rationale  for  the  later  TRL  date  of  this  mission, 
but  its  lower  price  does  bring  it  forward  in  the  sequence  relative  to  its  TRL  date. 

The  final  tier  includes  LIST,  PATH,  GRACE-11,  SCLP,  GACM,  and  3D-Winds.  These 
missions  are  sequenced  by  cost  with  no  exceptions. 

Using  the  cost  assumption  discussed  in  4. 1.2.1  to  infer  timing  (Equation  11),  this 
sequence  can  be  converted  into  a  schedule  (Table  16).  The  Decadal  survey  assumed  that 
the  annual  budget  would  return  to  the  FYOO  level  of  funding  of  approximately  $750M/year. 
Hence,  the  reference  schedule  fits  almost  entirely  within  a  decade. 
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Decadal  Survey  Reference  Case 

Tier 

Mission 

Launch  Date 

Tier  1 

2010-2013 

CLARREO 

2010.353 

GPSRO 

2010.553 

SMAP 

2010.953 

ICESAT-II 

2011.353 

DESDynl 

2012.287 

Tier  2 

2013-2016 

XOVWM 

2012.753 

HyspIRI 

2013.153 

ASCENDS 

2013.687 

SWOT 

2014.287 

GEO-CAPE 

2015.02 

ACE 

2016.087 

Tier  3 

2016-2020 

LIST 

2016.487 

PATH 

2017.087 

GRACE-II 

2017.687 

SCLP 

2018.353 

GACM 

2019.153 

3D-WINDS 

2020.02 

Table  16.  Reference  Case  Schedule 


Using  the  cost  assumption,  the  missions  loosely  stay  within  the  periods  of  their  intended 
tiers.  The  first  tier  completes  in  less  time  than  expected,  the  second  tier  takes  longer  than 
planned,  and  the  third  tier  is  matches  its  projection. 

This  schedule  can  be  combined  with  the  information  regarding  mission  values,  as 
derived  in  section  3.3,  to  depict  the  accrual  of  benefit  over  time  (Figure  38).  The  horizontal 
axis  express  time  and  the  vertical  axis  express  the  percentage  of  weighted  value  relative  to 
the  entire  campaign.  Although  benefit  is  actually  realized  at  the  time  of  launch,  this  plot 
illustrates  accrual  at  the  decision  point  (the  time  when  the  step  function  costing  profile 
begins);  hence,  the  decade  begins  with  the  value  of  CLARREO  already  counted,  even  though 
it  does  not  launch  until  the  second  quarter  of  2010. 
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Figure  38.  Cumulative  Benefit  over  Time 

The  algorithm  used  by  the  Decadal  Survey  is  reflected  in  the  time  between  mission 
launches — the  cheaper  missions  fly  first  within  each  tier,  highlighting  the  tier  breaks  (such 
as  that  between  DESDyni  and  XOVWM).  Although  this  plot  clarifies  the  weighted 
contributions  of  each  panel  to  stakeholders,  it  is  difficult  to  distinguish  benefit  profiles  of 
individual  panels. 

Panel-level  benefit  trends  are  highlighted  by  plotting  the  relative  accrual  of  benefit 
over  time  (Figure  39).  in  this  plot  the  benefit  gained  by  each  panel  is  normalized — every 
panel  starts  at  0%  and  ends  with  100%  of  its  value  for  the  campaign.  The  visualization 
makes  clear  the  differences  technology  readiness  makes  in  benefit  accrual.  The  Climate 
and  Water  panels  benefit  the  most  from  the  first  tier,  whereas  the  Weather,  Human  Health 
and  Security,  and  Land-use  panels  gain  the  most  in  the  second.  Additionally,  the  Weather 
panel  consistently  lags  behind  all  other — it  relies  the  most  on  later  TRL  missions.  The 
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outlier  is  the  Solid  Earth  panel — because  not  every  objective  can  be  satisfied  with  these  17 
missions,  it  never  actually  reaches  100%  satisfaction. 


Figure  39.  Relative  Panel  Benefit  Profile  for  Reference  Schedule 


43.2.2  The  Constrained  Reference  Case 

The  reference  case  schedule  is  composed  using  only  the  cost  constraint  and  the 
sequence  proposed  by  the  Decadal  Survey.  The  schedule  enumerated  in  Table  16 
noticeably  violates  the  TRL  constraint:  although  missions  are  ordered  by  tier,  the  actual 
readiness  date  is  ignored.  Applying  the  TRL  constraint  results  in  the  "Constrained 
Reference"  schedule  (Figure  40).  This  approach  forces  missions  with  later  TRL  dates  to 
launch  later  on  within  a  tier;  tier  boundaries  are  still  respected. 
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ASCENDS 

2013 

2013.687 

SWOT 

2013 

2014.287 

GEO-CAPE 

2015 

2015.02 

ACE 

2015 

2016.087 

PATH 

2015 

2017.087 

GRACE-II 

2016 

2017.687 

SCLP 

2016 

2018.353 

GACM 

2017 

2019.153 

3D-WINDS 

2016 

2020.02 

Constrained  Reference 

fission 

Sequence 

rRL+ Budget 
Based  Launch 
Date 

CLARREO 

2010.353 

ICESAT-II 

2010.753 

DESDynl 

2011.687 

^iPSRO 

2011.887 

^IMAP 

2012.287 

2012.753 

ASCENDS 

2013.287 

SWOT 

2013.887 

HysplRl 

2014.287 

GEO-CAPE 

2015.02 

ACE 

2016.087 

PATH 

2016.687 

GRACE-II 

2017.287 

LIST 

2017.687 

SCLP 

2018.353 

GACM 

2019.153 

3D-WINDS 

2020.02 

Figure  40.  Constrained  Reference  Derivation 

However,  the  constrained  reference  case  also  violates  the  strict  TRL  dates  by 
scheduling  missions  before  they  are  ready.  This  occurs  only  when  there  are  no  other 
possibilities:  hence  XOVWM,  the  first  of  the  second  tier  missions,  launches  in  2012  rather 
than  waiting  until  its  TRL  date  in  2013  to  fly,  as  there  are  no  other  missions  with  lower  TRL 
dates. 

The  constrained  reference  pushes  four  missions  backwards  in  the  campaign 
sequence:  GPSRO,  SMAP,  HysplRl,  and  LIST.  Although  these  missions'  relatively  lower 
costs  prioritize  them  within  a  tier,  they  are  limited  by  the  actual  dates  associated  with  the 
schedule.  The  accrual  of  benefit  for  this  schedule  can  then  be  compared  against  that  of  the 
reference  case  (Figure  41). 
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Figure  41.  Benefit  Profile  Comparison 


The  additional  constraint  subtly  changes  the  benefit  profile.  In  the  unconstrained  case 
missions  were  much  closer  to  being  arranged  by  cost  within  a  tier — with  the  TRL 
constraint  implemented  the  pattern  is  still  evident,  but  each  tier  tends  to  have  two 
arrangements  rather  than  one,  reflecting  the  two  TRL  dates  in  each  tier  as  evidenced  in 
Table  15. 

As  with  the  reference  case,  the  relative  benefit  profile  of  the  constrained  reference 
schedule  can  be  plotted  (Figure  42).  This  rescheduling  does  not  result  in  significant 
changes  to  the  patterns  seen  in  the  reference  case,  since  the  TRL  restrictions  only 
rearrange  missions  within  a  tier. 
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Constrianed  Reference  Case:  Relative  Benefit  over  Time 
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Figure  42.  Relative  Panel  Benefit  Profile  for  the  Constrained  Reference  Schedule 


Of  the  classes  of  constraints  discussed  in  4.1,  the  constrained  reference  only 
incorporates  two:  cost  and  TRL.  The  scope  constraint  does  apply  to  scheduling  algorithms. 
The  data  continuity  value-function  has  limited  applicability:  missions  are  close  enough 
together  in  time  that  no  continuity  issues  addressed  in  the  Decadal  Survey  are  raised.  The 
issue  of  fairness  does  not  overtly  arise:  with  the  exception  of  the  unsatisfied  Ocean 
Bathymetry  objective,  all  six  panels  accrue  the  totality  of  their  respective  benefits. 

Despite  its  limited  constraints  and  slight  difference  from  the  reference  sequence,  the 
constrained  reference  case  more  accurately  presents  the  sequencing  algorithm  enumerated 
in  the  Decadal  Survey.  For  this  reason,  algorithm  development  for  rearchitecting  the 
Decadal  campaign,  as  explained  in  the  following  section,  is  baselined  against  the 
constrained  reference. 
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4.3.3  Enhanced  Algorithm  Development 

The  constrained  reference  case  presents  a  valid  schedule  because  of  the 
assumptions  inherent  in  the  Decadal  Survey,  primarily  that  the  mission  costs  and  program 
budget  will  allow  the  completion  of  the  campaign  within  a  decade.  An  enhanced  algorithm 
is  necessary  to  deal  with  cases  when  missions  can  be  descoped  or  cancelled,  and  the  budget 
severely  limits  the  timing  of  missions.  This  section  outlines  the  process  used  to  define  a 
desirable  algorithm  for  campaign  scheduling  given  uncertainty.  First,  the  metrics  used  for 
selection  and  the  options  for  analysis  are  defined.  Then,  the  results  of  three  sensitivity 
experiments  are  presented.  Finally,  the  selected  algorithm  is  summarized. 

4.3.3. 1  Metrics  for  Evaluation  and  Algorithm  Options 

The  fitness  of  campaign  scheduling  algorithms  was  determined  using  three  metrics: 
closeness  to  the  constrained  reference,  normalized  undiscounted  benefit,  and  normalized 
discounted  benefit. 

1.  Closeness  to  the  constrained  reference  (years):  Every  campaign  schedule 
will  assign  a  particular  launch  date  to  each  mission.  The  sum  of  the  absolute 
differences  in  launch  dates  between  the  constrained  reference  schedule  and 
the  algorithmic  schedule  is  a  measure  of  how  "close"  the  sequence  of  the 
enhanced  algorithm  schedule  is  to  the  constrained  reference  sequence. 

2.  Normalized  un discounted  benefit  (percent  of  total  value):  This  metric 
represents  an  algorithm's  propensity  to  pick  high  value  missions.  It  is 
calculated  by  summing  the  undiscounted  benefit  accrued  by  each  mission, 
and  then  normalizing  this  value  by  time.  For  resequences  of  the  baseline 
Decadal  missions  this  metric  will  not  be  useful,  as  every  mission  will  be 
flown. 

3.  Normalized  discounted  benefit  (percentage  of  total  value):  This  metric 
reflects  an  algorithms  ability  to  arrange  missions  in  an  optimal  manner, 
minimizing  discounting  as  discussed  in  4.2. 
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The  constraints  discussed  in  4.1  and  the  instrument  benefit  contributions  discussed 
in  3.3  present  several  options  for  campaign  algorithm  criteria.  The  algorithm  experiments 
described  in  the  following  sections  explain  the  evaluations  of  the  following  five  factors: 

1.  TRL:  The  TRL  constraint  applied  as  a  limitation  to  the  earliest  launch  date  of 
a  mission,  as  described  for  the  constrained  reference  in  4. 3. 2. 2. 

2.  Cost:  Cost  applied  as  a  preference  for  scheduling  lower-cost  mission  first,  as 
described  for  the  constrained  reference  in  4.3. 2. 2. 

3.  Benefit:  Prefer  missions  with  higher  traceable  benefits,  as  outlined  in 
section  3.3. 

4.  Value:  Prefer  missions  with  higher  traceable  benefit  per  cost  rations,  as 
described  in  section  3.3. 

5.  Fairness:  Minimize  relative  benefit  accrual  deviation  as  discussed  in  4. 1.2. 4. 

The  algorithm  proposed  by  the  Decadal  Survey  depends  on  cost  and  TRL  for 
sequencing.  The  algorithm  proposed  by  Colson  relies  primarily  on  TRL  and  fairness  for 
sequencing.  Hence,  it  was  assumed  that  the  TRL  constraint  would  be  applied  in  every 
algorithm.  The  experiments  described  in  the  next  section  were  conducted  to  evaluate 
combinations  of  these  options  with  respect  to  the  metrics  described  above. 

4.33.2  Algorithm  Sensitivity  Analysis 

Three  experiments  were  conducted  to  evaluate  the  options  described  above.  The 
first  experiment  studied  variations  of  fairness  as  an  algorithm  parameter.  The  second 
experiment  explored  the  fitness  of  algorithm  options  with  respect  to  the  constrained 
reference.  The  final  experiment  was  used  to  tune  the  parameters  of  the  algorithm  in  a  less 
restrictive  scenario. 

First,  the  effectiveness  of  various  fairness  options  were  considered.  Using  the 
scenario  parameters  in  Table  17,  six  algorithms  were  used  to  generate  campaigns.  This 
scenario  replicates  the  Decadal  Survey  assumptions  for  cost  and  budget: 
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First  Algorithm  Experiment  Scenario  (Decadal  Baseline) 

Mission  parameters 

Scenario  parameters 

Name 

Cost 

TRL  date 

parameter 

value 

CLARREO 

265 

2010 

Annual  budget  ($M/yr) 

750 

GPSRO 

150 

2012 

Number  of  missions  to  be  flown 

17 

SMAP 

300 

2012 

ICESAT-II 

300 

2010 

DESDynl 

700 

2010 

XOVWM 

350 

2013 

HyspIRI 

300 

2015 

ASCENDS 

400 

2013 

SWOT 

450 

2013 

GEO-CAPE 

550 

2015 

ACE 

800 

2015 

LIST 

300 

2017 

PATH 

450 

2015 

GRACE-II 

450 

2016 

SCLP 

500 

2016 

GACM 

600 

2017 

3D-WINDS 

650 

2016 

Table  17.  First  Algorithm  Experiment  Scenario 


The  results  of  the  experiment  can  be  found  in  (Table  18).  The  general  trend  this 
experiment  revealed  is  that  the  more  specific  the  fairness  criteria  used,  the  less  valuable 
the  campaign  architecture  will  be.  The  algorithms  that  considered  objectives  sacrificed 
significant  value  for  the  sake  of  fairness,  and  it  was  not  clear  this  was  necessary.  The 
algorithm  that  sought  to  minimize  the  benefit  deviation  of  either  of  the  two  least  satisfied 
panels  was  selected  for  further  simulations.  While  the  campaign  the  2-panel  algorithm 
produced  was  not  as  valuable  as  the  1-panel  option,  it  was  within  a  reasonable  range. 
Allowing  the  algorithm  to  pick  missions  that  contribute  to  two  most  unsatisfied  panels 
opens  up  the  solutions  space  and  enables  the  algorithm  to  deal  with  situations  when  TRL 
does  not  allow  any  missions  for  the  least  satisfied  panel. 
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Minimize  the  Fairness  Deviation  of... 

Campaign  Discounted  Value 

The  Sum  of  Every  Panel 

0.640013 

Only  the  least  satisfied  panel 

0.641255 

Either  of  the  two  least  satisfied  panels 

0.639711 

Only  the  least  satisfied  objective 

0.626557 

Either  of  the  two  least  satisfied  objectives 

0.628445 

Any  of  the  three  least  satisfied  objectives 

0.633817 

Table  18.  Fairness  Experiment  Results 


Using  the  same  scenario  parameters,  a  second  algorithm  experiment  was  conducted 
analyzing  other  algorithm  criteria  with  respect  to  the  constrained  reference.  A  total  of  16 
algorithms  were  assembled  exploring  various  strengths  of  the  options  discussed  in  4.3.3. 1. 
Campaigns  were  assembled  using  these  algorithms  and  the  scenario  parameters  in  Table 
17,  and  were  subsequently  evaluated  with  respect  to  the  constrained  reference  and 
depreciated  value.  Three  algorithms  were  identified  to  be  on  the  Pareto  frontier  and  were 
selected  for  further  study  (Table  19). 


Algorithm  Option: 

1 

2 

3 

Depreciated  Value: 

64.31% 

64.38% 

64.92% 

Constraint: 

TRL 

TRL 

TRL 

Strong  Criteria: 

cost 

cost 

value 

Weak  Criteria: 

fairness 

benefit 

fairness 

1 

CLARREO 

CLARREO 

CLARREO 

2 

ICESAT-II 

ICESAT-II 

ICESAT-II 

3 

DESDynl 

DESDynl 

DESDynl 

4 

GPSRO 

GPSRO 

GPSRO 

5 

SMAP 

SMAP 

SMAP 

6 

XOVWM 

XOVWM 

XOVWM 

7 

ASCENDS 

ASCENDS 

ASCENDS 

8 

SWOT 

SWOT 

SWOT 

9 

HyspIRI 

HyspIRI 

HyspIRI 

10 

GEO-CAPE 

GEO-CAPE 

ACE 

11 

ACE 

ACE 

GEO-CAPE 

12 

PATH 

PATH 

GRACE-II 

13 

GRACE-II 

GRACE-II 

PATH 

14 

LIST 

LIST 

LIST 

15 

SCLP 

GACM 

GACM 

16 

GACM 

SCLP 

3D-WINDS 

17 

3D-WINDS 

3D-WINDS 

SCLP 

Differences  from  constrained  reference  marked  in  red 

Table  19.  Viable  Algorithm  Sequences  in  Baseline  Scenario 
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The  campaign  sequences  produced  by  these  algorithms  are  not  significantly  different  from 
each  other:  the  first  algorithm  identically  replicates  the  constrained  reference,  the  second 
option  reverses  the  order  of  one  pair  of  missions,  and  the  third  rearranges  three  sets  of 
missions. 

The  sensitivities  of  the  strong  and  weak  criteria  for  these  three  algorithms  were 
analyzed  with  the  final  experiment.  Scenario  parameters  were  chosen  that  did  not  allow 
the  completion  of  a  campaign  in  one  decade,  and  included  already  flown  missions  (Table 
20).  An  updated  version  of  this  scenario  is  discussed  in  Chapter  5. 


Second  Algorithm  Experiment  Scenario 

Mission  parameters 

Scenario  parameters 

Name 

Cost 

TRL  date 

parameter 

value 

CLARREO 

579 

2010 

Annual  budget  ($M/yr) 

300 

GPSRO 

230 

2012 

Number  of  missions  to  be 

flown 

8 

SMAP 

393 

2012 

Other  constraints: 

The  two  NOAA 

missions  were 
not  eligible  for 
scheduling 

The  SMAP  and 

ICESAT-II 

missions  have 

fixed  launch 

dates 

ICESAT-II 

607 

2010 

DESDynl 

1500 

2010 

XOVWM 

538 

2013 

HyspIRI 

500 

2015 

ASCENDS 

500 

2013 

SWOT 

800 

2013 

GEO-CAPE 

1276 

2015 

ACE 

1627 

2015 

LIST 

600 

2017 

PATH 

800 

2015 

GRACE-II 

500 

2016 

SCLP 

600 

2016 

GACM 

1030 

2017 

3D-WINDS 

800 

2016 

Table  20.  Second  Algorithm  Experiment  Scenario 


The  results  of  the  sensitivity  analysis  are  plotted  below  (Figure  43).  The  metrics 
used  for  evaluation  are  the  normalized  discounted  benefit  and  the  normalized 
undiscounted  benefit.  In  this  plot  the  utopia  point  is  depicted  in  the  upper-right  corner, 
pointing  to  the  right:  the  ideal  campaign  will  include  100%  of  the  possible  benefit 
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enumerated  in  the  CSTM  (by  including  all  seventeen  missions  within  a  decade),  and  will  be 
sequenced  such  that  depreciation  is  minimized,  although  the  depreciated  value  outweighs 
the  nondepreciated  value. 


Figure  43.  Second  Algorithm  Experiment:  Screening  Variations 

Based  upon  this  analysis,  the  final  algorithm  selected  is  represented  by  the  second 
point  to  the  right  of  the  plot,  "TRL>fairness>cost".  This  algorithm  utilizes  the  strong 
criteria  of  cost  and  the  weak  criteria  of  fairness  to  the  two  least  satisfied  panels.  While  it 
does  not  represent  the  optimal  discounted-benefit  solution,  it  does  implement  the  concept 
of  fairness  for  a  relatively  low  cost. 
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4.3.33  Final  Algorithm  Summary 

The  algorithm  that  will  be  used  in  Chapter  5  to  explore  rearchitecting  the  Decadal 
campaign  thus  closely  resembles  the  programmatic  decision  strategy  proposed  in  the 
Decadal  Survey — scheduling  on  budgetary  and  technological  risks  factors,  with  the 
addition  of  fairness.  The  following  outlines  a  simplified  explanation  of  the  algorithm: 

1.  First,  the  algorithm  searches  the  list  of  available,  un-flown  missions. 
Although  long-term  campaign  planning  should  incorporate  the  possibility  of 
the  operationalization  of  missions,  for  a  first  pass  this  algorithm  assumes 
each  mission  will  only  fly  once.  This  is  also  a  necessary  assumption  given 
that  the  CSTM  value  calculations  were  all  dependent  on  flying  each  Decadal 
mission  only  once.  This  is  described  in  4.1.3. 

2.  Second,  the  algorithm  pares  the  list  of  missions  down  to  those  that  are 
technologically  ready  given  the  date,  if  no  missions  are  available,  as  seen  in 
the  baseline  Decadal  scenario,  the  algorithm  pares  the  list  down  to  those 
missions  in  the  next  bin  of  TRL  dates — simulating  the  acceleration  of  the 
most  ready  technologies.  This  constraint  is  described  in  4. 1.2. 2. 

3.  Third,  the  algorithm  determines  which  two  panels  are  least  satisfied,  it 
does  this  by  computing  the  percentage  of  each  panel's  weighted  benefit  that 
has  been  accrued  relative  to  that  panel's  stakeholder  weighting.  Hence  the 
least  satisfied  panel  is  not  necessarily  the  panel  with  the  most  unfulfilled 
absolute  benefit.  This  process  is  described  in  4. 1.2.4. 

4.  Fourth,  the  algorithm  determines  which  missions  satisfy  the  two  least 
satisfied  panels,  if  there  are  no  missions  current  available  to  fly  that  meet 
this  restriction,  the  algorithm  expands  the  field  to  missions  that  satisfy  the 
top  three  panels,  and  so  forth  until  at  least  one  mission  meets  the  criteria. 

No  preference  is  given  to  missions  that  are  more  effective  in  satisfying  a 
panel-either  a  mission  does  or  does  not. 

5.  Finally,  from  the  missions  that  have  passed  through  all  the  previous  steps, 
the  algorithm  selects  the  lowest  costing  option,  if  two  missions  that  reach 
the  final  algorithm  step  are  of  the  same  price,  then  the  algorithm  picks  the 
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one  best  satisfies  both  of  the  least  satisfied  panels.  If  the  missions  are 
qualitative  identical,  the  number  of  panels  included  in  this  calculation  is 
expanded  (by  panel  dissatisfaction  order)  until  a  difference  is  found.  This 
mission  is  then  added  to  the  schedule,  and  time  is  advanced  using  the 
cost/budget  assumption. 

This  algorithm  was  validated  against  the  Decadal  Survey  by  applying  it  to  the  first 
experiment  scenario  described  in  this  chapter  (Table  17).  The  schedule  produced  by  the 
final  CSTM  algorithm  was  then  compared  against  the  constrained  reference  case  (Table 
21).  The  results  were  identical:  given  the  constraints  inherent  in  the  original  Decadal 
Survey,  the  final  CSTM  algorithm  can  replicate  the  Decadal  campaign  (Table  21). 


Constrained  Reference 


CSTM  Algorithm 


SWOT 


HyspIRI 

GEO-CAPE 

ACE 

PATH 

GRACE-II 


UST 

sap 

GACM 


3D-WINDS 


^CM 

3D-WINDS 


Mission 

TRL+Budet  Based 
Launch  Date 

Mission 

TRL+Fairness+Bud 
get  Based  Launch 
Date 

CLARREO 

2010.353 

CLARREO 

2010.353 

ICE5AT-II 

2010.753 

ICESAT-II 

2010.753 

DESDynl 

2011.687 

DESDynl 

2011.687 

GPSRO 

2011.887 

GPSRO 

2011.887 

SMAP 

2012.281 

KOVWM 

2012.7M 

ASCENDS 

2013.281 

II 

2013.887 

2014.287 

2015.02 

2016.087 


2016.687 
2017.287 

2017.687 
2018.353 
2019.153 

2020.02 


Table  21.  Final  Algorithm  Applied  to  Baseline  Scenario 


4.4  Summary 

This  chapter  described  the  use  of  the  CSTM  for  campaign  design.  Constraints  were 
applied  to  the  CSTM  based  upon  guidance  from  the  Decadal  Survey  and  Colson's  thesis. 
Value  functions  relevant  to  total  benefit,  present  value,  and  data  continuity  were  described. 
The  Decadal  Survey  was  used  to  inform  a  reference  algorithm.  An  improved  algorithm  was 
developed  and  validated  against  the  reference  algorithm.  This  algorithm  will  be  used  in  the 
following  chapter  to  explore  campaign  planning  with  post-Decadal  assumptions. 
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5  Scheduling  Simulation  Results 


The  CSTM  scheduling  algorithm  has  been  validated  against  the  original  Decadal 
Survey  assumptions.  The  section  presents  the  results  of  a  series  of  scheduling  simulations 
that  apply  current  assumptions  to  the  Decadal  campaign:  the  new  mission  costs,  the 
reduced  annual  budget,  the  loss  and  degradation  of  precursor  missions,  and  the  addition  of 
international  missions,  as  discussed  in  Section  1.1. 

Objective  of  Scheduling  Simulation  Results 

•  To  examine  the  impacts  of  key  changes  that  have  occurred  since  the  publishing  of 
the  Decadal  Survey  by  attaining  updated  assumptions  using  surveys  of  NASA 
scientists  and  Engineers 

The  primary  problem  addressed  by  this  chapter  is  the  issue  of  cost  growth.  The 
strategy  of  breaking  apart  the  original  Decadal  survey  missions  into  single-instrument 
platforms  is  presented  as  a  possible  solution.  Variations  of  campaign  parameters  and  a 
sensitivity  analysis  are  presented  demonstrating  the  utility  of  this  approach.  This  chapter 
is  organized  into  the  following  sections: 

•  Section  5.1:  Changes  to  the  Decadal  sequence  based  upon  new  assumptions. 

This  section  presents  a  comparison  between  the  constrained  reference  sequence 
and  a  campaign  scheduled  with  updated  assumptions. 

•  Section  5.2:  Rescheduling  the  Decadal  campaign  with  reassigned  instruments. 
This  section  presents  a  comparison  between  scheduling  the  originally  proposed 
Decadal  missions  and  scheduling  the  instruments  of  the  Decadal  campaign  as 
individual  missions. 

•  Section  5.3:  Completing  the  NASA  Schedule.  This  section  examines  scheduling  a 
limited  subset  of  missions  given  the  campaign  decisions  already  made  by  NASA. 

•  Section  5.4:  Budget  Sensitivity.  This  section  analyzes  the  sensitivity  of  a 
campaign  to  budgetary  changes. 

•  Section  5.5:  Summary. 


112 


5.1  Changes  to  the  Decadal  Sequence  Based  upon  New  Assumptions 


5.1.1  Motivation 

Chapter  4  presented  a  justification  for  the  CSTM  algorithm,  which,  given  the  original 
Decadal  assumptions,  could  reproduce  the  constrained  reference  sequence.  This  section 
presents  the  application  of  the  CSTM  algorithm  to  scheduling,  given  updated  assumptions, 
and  an  analysis  of  how  they  affect  the  sequencing  of  Decadal  missions. 

5.1.2  Parameters 

For  this  simulation  the  constrained  reference  sequence  was  compared  to  a 
campaign  scheduled  using  the  CSTM  algorithm  and  recosted  missions.  The  scenario 
parameters  found  in  Table  22  were  assumed.  The  costs  and  TRL  dates  for  both  schedules 
can  be  found  in  Table  23.  The  results  of  this  simulation  are  compared  to  the  constrained 
reference  sequence  presented  in  Table  21  of  section  4.3. 3. 3. 


Campaign 

Parameter 

Constrained 

Reference 

Campaign 

Recosted  Decadal 

Mission  Set 

Annual  Budget 

$750  M 

$300  M 

Mission  Costs 

Decadal  Survey 

NASA  estimates 
5/09 

Instrument  TRLs 

Decadal  Survey 

NASA  estimates 
5/09 

Science  Panel 

Total  Value 

Discount  Rate 

Weather 

0.214 

0.10 

Climate 

0.206 

0.15 

Ecosystems 

0.206 

0.10 

Water 

0.156 

0.10 

Health 

0.111 

0.10 

Solid  Earth 

0.107 

0.05 

Table  22.  New  Assumption  Scenario  Parameters 


5.1.3  Results 

Table  23  displays  how  each  mission  changed  in  the  sequence  with  updated 
assumptions.  The  different  colors  indicate  the  Decadal  tiers.  A  new  set  of  tiers,  "A,  B,  and 
C"  are  indicated  for  the  updated  campaign. 
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Launch  Date 

Reference 

Cost 

TRL  Date 

Constrained 

[feference 

Revised 

Assumption 

Reference 

TRL  Date 

Revised 

Cost 

Launch  Date 

2010.4 

265 

2010 

CLARREO 

GPSRO 

2010 

230 

2010.8 

2010.8 

300 

2010 

ICESAT-II 

GRACE-II 

2009 

500 

2012.4 

V 

2011.7 

700 

2010 

DESDynl 

HyspIRI 

2012 

500 

2014.1 

41 

1— 

2011.9 

150 

2012 

GPSRO 

ASCENDS 

2013 

500 

2015.8 

1- 

2012.3 

300 

2012 

SMAP 

KOVWM 

2010 

538 

2017.6 

2012.8 

350 

2013 

KOVWM 

CLARREO 

2012 

579 

2019.5 

2013.3 

400 

2013 

ASCENDS 

SMAP 

2009 

450 

2021.0 

Ol 

2013.9 

450 

2013 

SWOT 

LIST 

2015 

600 

2023.0 

m 

41 

1- 

2014.3 

300 

2015 

ityspIRI 

SCLP 

2011 

600 

2025.0 

t- 

2015.0 

550 

2015 

GEO-CAPE 

3D-WINDS 

2012 

800 

2027.7 

2016.1 

800 

2015 

ACE 

PATH 

2017 

800 

2030.3 

2016.7 

450 

2015 

PATH 

ICESAT-II 

2010 

850 

2033.2 

2017.3 

450 

2016 

GRACE  II 

2014 

1030 

2036.6 

cn 

2017.7 

300 

2017 

UST 

GEO-CAPE 

2014 

1276 

2040.8 

o 

41 

1- 

2018.4 

500 

2016 

SCLP 

SWOT 

2011 

800 

2043.5 

41 

1- 

2019.2 

600 

2017 

GACM 

/  \ 

DESDynl 

2012 

1500 

2048.5 

2020.0 

650 

2017 

3D-WINDS 

ACE 

2013 

1800 

2054.5 

Table  23.  Sequence  Comparison 


The  first  tier  of  Decadal  missions  spreads  out  evenly  across  the  entire  campaign, 
with  GPSRO  becoming  the  first  mission  scheduled,  CLARREO  and  SMAP  beginning  tier  "B", 
and  lCESat-11  and  DESDynl  falling  to  tier  "C".  The  second  tier  of  missions  congregates 
primarily  in  tier  "A",  although  the  more  expensive  SWOT,  GEO-CAPE,  and  ACE  missions 
move  to  tier  "C".  The  third  tier  of  missions  mostly  moves  forward  to  occupy  tier  "B",  with 
the  GRACE  mission  notably  moving  to  second  in  the  queue. 

5.1.4  Interpretation 

The  first  explanation  for  this  result  is  that  TRL  is  not  an  active  constraint.  In  the 
constrained  reference  case,  the  TRL  dates  of  each  mission  ensure  that  a  diverse  cross- 
section  of  mission  sizes  is  scheduled  in  each  tier.  The  original  TRL  dates,  however,  are  not 
traceable  exclusively  to  instrument  readiness — they  incorporate  data  continuity 
assumptions  as  well.  Additionally,  the  TRL  dates  assigned  to  missions  a  posteri  are 
functions  of  the  campaign  sequence:  missions  scheduled  to  fly  in  a  decade  do  not  need  to 
be  developed  right  now,  and  hence  have  alter  readiness  dates.  The  revised  TRL  dates 
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instead  assume  that  if  a  mission  is  prioritized  first  in  the  sequence,  that  date  will  be  the 
earliest  it  will  be  ready  to  fly.  The  latest  of  the  revised  TRL  dates,  2017,  which  is  associated 
with  the  more  expensive  PATH  mission.  Because  this  constraint  is  not  active,  the  primary 
criterion  for  scheduling  is  prioritization  based  upon  cost. 

The  missions  are  arranged  almost  exactly  in  increasing  cost  order.  This  is  reflective 
of  the  algorithms  preference  for  low  budget  risk  missions.  However,  two  exceptions  occur 
in  the  schedule:  the  SMAP  and  SWOT  missions.  This  is  evidence  of  the  fairness  criteria 
being  applied,  as  seen  by  the  plot  of  the  relative  nondiscounted  value  accrual  over  time 
(Figure  44). 


Figure  44.  Fairness  for  New  Assumption  Campaign 

At  most  points  in  time  the  Water  panel  is  the  most  satisfied.  A  schedule  with  only 
cost-ordering  would  fly  SMAP  second;  however,  because  the  CSTM  algorithm  includes 
fairness,  and  SMAP  is  primarily  a  Water  mission,  it  is  moved  further  back  in  the  queue  . 
Hence,  it  moves  to  seventh  in  the  sequence,  by  which  time  the  Health  panel,  the  only  other 
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recipient  of  SMAP  benefit,  is  one  of  the  two  least  satisfied  missions.  This  effect  is  mirrored 
in  the  scheduling  of  SWOT,  which  delivers  value  to  both  Water  and  Solid  Earth  panels. 

This  comparison  shows  that  there  may  be  an  unstated  utility  to  maintaining  the 
Decadal  survey  tiers.  Although  it  recommended  only  TRL  and  Cost  as  decision  criteria, 
even  with  the  added  conception  of  fairness,  the  Decadal  tiers  are  not  recoverable.  This 
comparison  highlights  the  need  for  further  algorithm  experimentation  and  development. 

5.2  Rescheduling  the  Decadal  Campaign  with  Reassigned  instruments 

5.2.1  Motivation 

With  the  current  mission  cost  and  budget  projections  it  will  take  more  than  40  years 
to  complete  the  Decadal  campaign.  The  scope  of  the  Decadal  Survey  was  exactly  that — a 
program  of  space  missions  that  pushed  the  boundaries  of  technology,  but  could  realistically 
be  achieved  in  the  next  decade.  A  new  approach  is  necessary  to  constrict  the  campaign  to  a 
reasonable  size. 

One  of  the  issues  presented  in  the  Decadal  Survey  is  that  of  mission  size.  Table  10  in 
Section  3.1.5  presented  the  number  of  instruments  in  each  mission,  highlighting  the 
correlation  between  number  of  instruments  and  mission  cost.  This  section  presents  a 
comparison  between  scheduling  the  original  Decadal  missions  and  scheduling  the  same 
instruments  reassigned  onto  unique  missions. 

Campaigns  were  scheduled  for  both  missions  sets  using  the  algorithm  identified  in 
the  previous  chapter.  Discounting  was  applied  to  the  value  of  both  campaigns  and  the 
resultant  impacts  were  analyzed. 

5.2.2  Parameters 

Two  campaigns  were  scheduled:  one  using  the  original  Decadal  mission  set  and  one 
using  the  reassigned  set.  The  baseline  scenario  parameters  are  similar  (Table  24).  Every 
mission  in  the  mission  set  was  flown,  and  no  time  limit  was  placed  on  the  campaign. 


116 


Campaign 

Parameter 

Decadal  Mission 

Set 

Free-flyer  Mission 
set 

Annual  Budget 

$300  M 

$300  M 

Mission  Costs 

NASA  estimates 
5/09 

NASA  estimates 
5/09 

Instrument  TRLs 

NASA  estimates 
5/09 

NASA  estimates 
5/09 

Science  Panel 

Total  Value 

Discount  Rate 

Weather 

0.214 

0.10 

Climate 

0.206 

0.15 

Ecosystems 

0.206 

0.10 

Water 

0.156 

0.10 

Health 

0.111 

0.10 

Solid  Earth 

0.107 

0.05 

Table  24.  Baseline  Scenario  Parameters 

Table  25  outlines  how  the  instruments  on  each  mission  were  separated.  This 
scenario  only  focused  on  splitting  apart  larger  missions  and  not  reassembling  different 
combinations,  in  this  diagram  the  color  of  each  instrument  indicates  if  it  was  separated 
into  an  additional  mission,  instruments  with  shared  colors  were  kept  together.  The  GPS 
receiver  for  CLARREO  was  assigned  to  both  sub-missions. 


CLARREO 

thermal-IR 

spectrometer/interfero 

meter 

GPS  receiver* 

Near  IR-VIS-UV 

spectrometer/interfero 

meter 

GPSRO 

advanced  GPS  receiver 

SMAP 

L-band  radar 

L-band  radiometer 

ICESAT-II 

modified  GLAS  (LIDAR) 

DESDynl 

L-band  inSAR 

IR  multi-beam  LIDAR 

XOVWM 

Ku  band  SAR 

scatterometer 

C-band  real-aperture 
scatterometer 

passive  (SRAD  with  K 
and  X  bands)  radiometer 

HyspIRI 

thermal  multispectral 
scanner  spectrometer 

Optical  Hyperspectral 
imager 

ASCENDS 

1.57or2.06um  LIDAR 

0.76  or  1.27  urn  LIDAR 

IR  radiometer 

SWOT 

Ku-band  radar  altimeter 

Ku-band  InSAR 

3-band  MW  radiometer 

GEO-CAPE 

steerable  hi-res 
spectrometer 

NIR/VIS/UV  wide-area 
spectrometer 

IR  correlation 

radiometer 

ACE 

cross-track  scanning 
cloud  radar 

multi-band  VIS/UV 
spectrometer 

Multi-beam  cross-track 
dual-wavelength  LIDAR 

Multi-angle  multi¬ 
wavelength  polarimeter 

LIST 

LIDAR  altimeter  with 
spatial  mapping 

PATH 

MW  spectrometer 

MW  radiometer 

GRACE-II 

Sat-to-Sat  ranger  and 
accelerometer 

SCLP 

dual  frequency  SAR 

passive  MW  radiometer 

GACM 

SWIR/IR  spectrometer 

MW  spectrometer 

UV/VIS  spectrometer 

UV/VIS  differential 
absorption  LIDAR 

3D-WINDS 

non-coherent  wind  Ildar 

Coherent  wind  lidar 

Table  25.  Simplified  Mission  Lysis 
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The  following  table  outlines  the  composition  of  each  mission  set  in  more  detail 
(Table  26).  The  Decadal  mission  set  includes  the  17  missions  described  in  the  Decadal 
Survey.  Each  of  these  missions,  displayed  vertically  on  the  table,  is  attributed  an  updated 
cost  and  TRL.  Additionally,  the  instrument  on  each  Decadal  mission  are  listed  (different 
colors  indicate  different  missions).  The  horizontal  mission  set  includes  the  reassigned 
satellites.  When  reasonable,  the  Decadal  survey  missions  were  broken  apart,  resulting  in 
26  new  missions.  Recombination  of  Decadal  instruments  was  not  analyzed.  Each  free  flyer 
includes  as  cost  estimate  provided  by  NASA,  as  well  as  a  TRL  date  reflective  of  the 
instruments  on  board. 
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300 

2012 

X 

X 

Clarreo  B 

300 

2012 

X 

X 

GPSRO 

230 

2010 

X 

SMAP 

450 

2009 

X 

X 

ICESAT-II 

850 

2010 

X 

DESDvnIA 

900 

2012 

X 

DESDvnl  B 

600 

2010 

X 

XOVWM 

538 

2010 

X 

X 

X 

HYSPIRIA 

350 

2010 

X 

HYSPIRI  B 

350 

2012 

X 

ASCENDS  A 

450 

2013 

X 

X 

ASCENDS B 

250 

2011 

X 

SWOT  A 

350 

2011 

X 

SWOTB 

650 

2011 

X 

X 

GEOCAPE A 

600 

2014 

X 

GEOCAPE  B 

400 

2013 

X 

GEOCAPE C 

400 

2013 

X 

ACE  A 

900 

2012 

X 

ACEB 

300 

2012 

X 

ACEC 

600 

2013 

X 

X 

USTA 

600 

2015 

X 

PATH  A 

800 

2017 

X 

X 

GRACE A 

500 

2009 

X 

SCLPA 

600 

2011 

X 

X 

GACMA 

1030 

2014 

X 

X 

X 

X 

WINDS  A 

800 

2012 

X 

X 

Table  26.  Mission  Set  Composition 
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Of  the  38  Instruments  originally  identified,  only  26  reassigned  missions  were 
created.  ICESat-II,  LIST,  GRACE-II,  and  GPSRO  were  not  separable,  as  they  only  had  one 
instrument  each.  The  NOAA  mission  XOVWM,  and  the  NASA  mission  GACM  were  not 
separated  due  to  lack  of  detailed  instrument  knowledge.  SMAP,  3D-Winds,  and  parts  of 
ASCENDS  were  not  separated  due  to  the  use  of  a  shared  component  by  two  instruments 
(SMAP’s  instruments  share  a  6m  rotating  antenna,  3D-WINDs  share  a  set  of  four 
telescopes).  PATH  and  SCLP,  and  parts  of  ACE  and  SWOT,  were  not  separated  due  to  need 
for  concurrent  measurements.  The  CLARREO  mission,  which  utilizes  three  separate 
instruments,  was  split  into  missions,  each  with  a  different  type  of  spectrometer  and  a 
shared  GPS  receiver. 

5.2.3  Results 

Table  27  lists  the  results  of  the  simulation.  The  left-hand  columns  of  the  table 
indicate  when  the  original  Decadal  missions  were  scheduled.  The  right-hand  columns  of 
the  table  indicate  when  the  corresponding  reassigned  missions  were  scheduled.  The  colors 
associated  with  the  reassigned  mission  are  indicative  of  the  differences  in  launch  date  from 
the  corresponding  Decadal  mission,  which  is  calculated  in  the  far  right  column.  Green 
indicates  the  smallest  difference,  followed  by  light  green,  yellow,  orange,  red,  dark  red,  and 
black.  The  absolute  total  difference  in  launch  dates  equals  276  years,  which  indicates,  on 
average,  a  10.6  year  deviation  from  the  Decadal  mission  launch  date. 
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Decadal 

Missions 

Launch 

Date 

Reassigned 

Missions 

Launch 

Date 

Difference 

(years) 

3D-WINDS 

2027.7 

WINDS  A 

2047.6 

19.9 

2054.5 

ACE  A 

2053.6 

-0.9 

2054.5 

ACE  B 

2014.8 

-39.7 

ACE 

2054.5 

ACE  C 

2033.7 

-20.8 

2015.8 

ASCENDS  A 

2035.2 

19.4 

ASCENDS 

2015.8 

ASCENDS  B 

2012.8 

-3 

2019.5 

CLARREO  A 

2013.8 

-5.7 

CLARREO 

2019.5 

CLARREO  B 

2015.8 

-3.7 

2048.5 

DESDynl A 

2050.6 

2.1 

DESDynl 

2048.5 

DESDynl B 

2031.7 

-16.8 

GACM 

2036.6 

GACM  A 

2057 

20.4 

2040.8 

GEOCAPE  A 

2037.2 

-3.6 

2040.8 

GEOCAPE  B 

2019.4 

-21.4 

GEO-CAPE 

2040.8 

GEOCAPE  C 

2027.7 

-13.1 

GPSRO 

2010.8 

GPSRO 

2010.8 

0 

GRACE-II 

2012.4 

GRACE  A 

2022.6 

10.2 

2014.1 

HYSPIRI  A 

2011.9 

-2.2 

HyspIRI 

2014.1 

HYSPIRI  B 

2018.1 

4 

ICESAT-II 

2033.2 

ICESAT-II 

2044.9 

11.7 

LIST 

2023 

LIST  A 

2026.4 

3.4 

PATH 

2030.3 

PATH  A 

2042.1 

11.8 

SCLP 

2025 

SCLP  A 

2029.7 

4.7 

SMAP 

2021 

SMAP 

2020.9 

-0.1 

2043.5 

SWOT  A 

2016.9 

-26.6 

SWOT 

2043.5 

SWOT  B 

2039.4 

-4.1 

XOVWM 

2017.6 

XOVWM 

2024.4 

6.8 

Total 

276.2 

Table  27.  Baseline  Results 


The  biggest  change  in  launch  dates  comes  from  the  ACE  mission.  As  presented  in 
Section  3.3,  the  ACE  mission  is  by  far  the  most  valuable.  However,  it  is  also  the  most 
expensive,  with  the  current  estimate  running  at  $1.8B.  The  CSTM  algorithm  will  attempt  to 
schedule  the  most  expensive  mission  last,  regardless  of  benefit  delivered.  Breaking  ACE 
apart,  however,  allows  the  cheaper,  yet  still  significantly  beneficial,  portions  to  fly  earlier 
on  in  the  campaign.  Conversely,  the  GACM  mission,  which  had  been  one  of  the  more 
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expensive  of  the  17  Decadal  missions,  became  the  most  expensive  mission  because  it  was 
not  reassigned,  and  was  subsequently  scheduled  last  in  the  campaign  by  the  CSTM 
algorithm. 

The  schedules  were  then  evaluated  with  respect  to  the  discounted  value  of  the 
campaign.  As  both  mission  sets  included  every  instrument  proposed  in  the  Decadal  Survey, 
the  total  non-discounted  value  for  both  campaigns  was  identical.  The  depreciation  of  value 
over  time  is  presented  in  Figure  45. 

In  this  figure  the  depreciation  of  a  mission  value  is  a  function  of  the  panel  discount 
rates  (Table  24)  and  the  value  of  the  mission  as  determined  through  the  CSTM.  The  black 
line  represents  the  cumulative  benefit  actually  captured  by  a  campaign — realized  when  a 
mission  launches.  Value  depreciation  stops  once  a  mission  has  been  launched,  hence  it  is 
desirable  that  the  scheduler  capture  as  much  benefit  as  possible  before  it  depreciates.  The 
length  of  time  between  steps  on  the  black  line  is  indicative  of  the  cost  of  the  mission,  and 
the  scheduler  preference  for  low-cost  mission  first  is  seen  in  the  elongated  steps  later  on  in 
both  campaigns. 
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Discounted  Value 
for  Baseline  Decadal  Missions 
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Figure  45.  Discounted  Value  over  Time 
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The  reassigned  mission  set,  although  more  expensive  overall,  is  more  effective  at 
accruing  benefit  quickly  (Figure  46).  The  biggest  separation  occurs  4  years  into  the 
campaign  when  the  reassigned  campaign  schedules  the  ACE-B  mission.  A  significant 
portion  of  ACE's  value  is  accrued  early  on  in  the  campaign,  as  opposed  to  the  Decadal 
campaign,  which  schedules  the  entire  ACE  mission  40  years  later. 


Depreciated  Value  over  Time 


Figure  46.  Comparison  of  Discounting 

The  key  difference  between  mission  sets  campaigns  are  summarized  in  Table  28. 
The  Decadal  mission  set  is  slightly  less  expensive,  and  its  campaign  will  complete  a  few 
years  earlier;  however  there  is  a  huge  difference  in  discounted  value.  The  reassigned 
campaign  accrues  as  much  value  in  9.5  years  as  the  Decadal  campaign  accrues  in  44.5. 


Discounted 

Campaign 

Value 

Final  Launch 

Date 

Total  Cost 
($M) 

Decadal 

Mission  Set 

0.22 

2054.5 

13353 

Reassigned 
Mission  Set 

0.31 

2057.0 

14098 

Table  28.  Simulation  Results  Summary 
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5.2.4  Interpretation 

The  reassigned  campaign  is  much  more  effective  because  breaking  apart  missions 
changes  their  cost  to  benefit  ratios  (Figure  47).  Missions  in  is  set  on  average  accrue  more 
benefit  per  dollar  than  missions  in  the  Decadal  set.  The  most  valuable  campaign  (the  least 
discounting)  will  schedule  the  high  benefit-to-cost  missions  first. 
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Figure  47.  Cost  to  Benefit 


The  algorithm  chosen  for  scheduling,  however,  does  not  take  into  account  the  benefit  of 
each  mission,  as  the  Decadal  Survey  had  no  explicit  system  for  enumerating  value  when  it 
designed  the  Decadal  campaign.  Breaking  the  campaign  into  a  series  of  smaller  missions 
allows  a  campaign  to  accrue  benefit  before  discounting  takes  a  significant  toll. 
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5.3  Completing  the  NASA  Campaign 


5.3.1  Motivation 

A  second  simulation  was  conducted  to  consider  rearchitecting  the  Decadal 
campaign  from  the  perspective  of  NASA  planners.  This  analysis  was  conducted  to  inform 
campaign  design  decisions  given  the  high-level  decisions  that  have  already  been  made. 

5.3.2  Parameters 

In  this  more  limited  experiment  the  same  set  of  mission  and  scenario  parameters 
described  in  the  previous  section  were  utilized  with  three  significant  changes: 

1.  First,  this  simulation  only  allowed  the  scheduling  of  the  15  NASA  missions 
and  not  the  two  NOAA  missions.  It  is  not  known  at  this  time  how  integrated 
NASA  and  NOAA  efforts  will  be  in  completing  the  Decadal  Survey;  hence  it  is 
assumed  that  the  NOAA  missions  will  be  handled  independently. 

2.  Second,  the  SMAP  and  lCESat-11  missions  are  already  assigned  launch  dates. 
Currently  these  two  missions  are  the  most  well-developed  and  have  been 
tentatively  scheduled  to  launch  in  2013  and  2015,  respectively.  For  this 
simulation  it  is  assumed  that  SMAP  will  launch  in  2013.5,  and  that  lCESAT-11 
will  launch  next,  with  the  exact  date  being  a  function  of  the  annual  budget,  as 
discussed  in  4. 1.2.1.  Once  again,  the  budget  is  assumed  to  be  $300M/year. 

3.  Third,  the  campaign  is  limited  to  a  20-year  span,  starting  in  2010.  No 
missions  are  scheduled  after  2030  because  it  is  unknown  if  NASA  has  the 
capability  for  such  far-horizon  planning. 

Because  of  these  parameters,  the  value  remaining  to  be  accrued  is  different  than  in 
previous  simulations  (Figure  48).  The  contributions  of  NOAA  to  the  entire  campaign  are 
represented  by  the  top  purple  area — this  is  not  eligible  for  NASA  campaign  planning.  The 
contributions  to  campaign  benefit  from  SMAP  and  lCESat-11  are  already  fixed,  as 
represented  by  the  black  line.  At  the  time  of  lCESat-11  launch  (6.3  years  into  the  campaign), 
only  45%  of  the  total  benefit  is  still  available  to  be  scheduled. 
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Discounted  Value 
for  NASA-only  Simulation 
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Figure  48.  Uncaptured  Benefit  at  start  of  NASA-only  simulation 
5.3.3  Results 

The  results  of  the  simulations  are  displayed  in  Table  29.  in  this  table,  missions  are 
divided  into  three  tiers:  those  that  were  fixed  in  the  simulation,  those  launching  between 
2017  and  2023,  and  those  launching  between  2024  and  2030.  The  fixed  missions  are 
highlighted  in  green.  Reassigned  missions  that  also  appear  in  the  original  mission  set 
schedule  are  boxed  in  pink.  Missions  that  are  not  represented  by  both  schedules  are 
depicted  in  gray. 


Reassigned  Mission  Set 

Original  Mission  Set 

Fixed 

SMAP 

2013.5 

SMAP 

2013.5 

ICESAT-II 

2016.3 

ICESAT-II 

2016.3 

2017-2023 

ASCENDS  B 

2017.2 

HyspIRI 

2018.0 

ACE  B 

2018.2 

ASCENDS 

2019.7 

CLARREO  A 

2019.2 

GRACE-II 

2021.3 

CLARREO  B 

2020.2 

CLARREO 

2023.3 

HYSPIRI  B 

2021.3 

HYSPIRI  A 

2022.5 

SWOT  A 

2023.7 

2024-2030 

GEOCAPE  B 

2025.0 

LIST 

2025.3 

GEOCAPE  C 

2026.3 

SCLP 

2027.3 

GRACE  A 

2028.0 

PATH 

2029.9 

ACE  C 

2030.0 

Table  29.  NASA-only  Campaign  Schedules 
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All  of  the  2017-2023  missions  from  the  Decadal  mission  set  are  at  least  partially 
represented  in  the  free-flyer  sequence.  The  only  instrument  missing  is  ASCENDS  A,  which 
contains  the  two  LiDAR  instruments  from  the  original  mission  concept.  None  of  the  2024- 
2030  missions  are  represented,  instead,  free  flyers  from  SWOT,  GEOCAPE,  and  ACE  are 
included  in  the  campaign.  Each  of  these  missions  were  originally  represented  in  the  tier 
"C"  of  the  original  Decadal  resequencing  (5.1.3,  Table  23). 

As  was  seen  in  the  previous  section,  a  comparison  of  the  discounted  value  of  each 
campaign  indicates  the  reassigned,  smaller  mission  set  is  more  valuable  (Table  30).  The 
depreciation  of  value  over  time  for  both  campaigns  can  be  found  in  Figure  49,  but  as  in  the 
previous  analysis,  the  high  benefit-to-cost  missions  of  the  reassigned  campaign  make  it 
better  suited  to  accruing  value.  This  is  particularly  evident  in  the  scheduling  of  the 
originally  large  ACE  and  GEOCAPE  components  early  on — the  Decadal  missions  provide 
significant  benefit,  more  of  which  can  be  realized  when  the  missions  are  split  into  smaller 
pieces. 


NASA-only  campaign 

Discounted  Campaign 
Value 

Final  Launch 

Date 

Total  Cost 
($M) 

Decadal  Mission 

Set 

0.143 

2029.9 

5379 

Reassigned  Mission 
Set 

0.184 

2030.0 

5400 

Table  30.  NASA-only  Results 
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Figure  49.  NASA-only  Discounted  Value  over  Time 
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5.3.4  Interpretation 

The  results  in  Table  29  indicate  that  there  are  a  number  of  missions  that  should  be 
flown  first.  The  HyspiRi,  ASCENDS,  GRACE-ii,  and  CLARREO  missions  all  deliver  value  at 
lower  costs.  Additionally,  larger  missions  such  as  GEOCAPE  and  ACE  should  be  considered 
for  decomposition — their  instruments  are  too  valuable  to  tie  up  in  budget  risk  factors  and 
schedule  at  the  end  of  the  campaign. 

5.4  Budget  Sensitivity 

5.4.1  Motivation 

As  a  final  analysis,  the  sensitivity  of  the  NASA-only  free-flyer  campaign  to  budget 
variations  was  analyzed.  Understanding  the  impacts  of  changes  enables  dialog  between  the 
campaign  architect  and  policy  makers. 

5.4.2  Parameters 

The  scenario  parameters  discussed  in  the  previous  section  were  applied  with  the 
following  exception: 

•  The  campaign  was  not  limited  to  a  20  year  duration 

For  this  simulation  changing  the  budget  did  not  actually  change  the  sequence  of 
missions — only  the  timing  and  depreciated  value.  Hence,  only  one  sequence  is  presented. 

5.4.3  Results 

The  results  of  the  simulation  are  presented  in  Table  31.  in  this  table  the  sequences 
of  free-flyer  missions  is  presented  on  the  left.  Each  mission  is  mapped  to  an  annual  budget 
amount  varying  from  $300-1000M/year.  The  percentage  located  in  each  box  is  the 
cumulative  discounted  value  that  has  been  captured  at  the  time  of  that  particular  missions 
launch  for  that  given  budget.  Additionally,  the  color  scheme  represents  five-year 
increments  in  the  actual  schedule:  dark  green  missions  fly  within  the  first  5  years  of  2010, 
light  green  within  10  and  so  forth  through  black  missions,  which  fly  within  45  years. 
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Table  31.  Budget  Sensitivity 


5.4.4  Interpretation 

Budget  plays  a  huge  role  in  the  actual  benefit  accrued  by  each  mission,  in 
comparing  the  original  mission  set  to  the  free-flyer  set  in  section  5.2  it  was  noted  that  in 
less  than  ten  years  the  free-flyer  campaign  accrued  as  much  value  as  the  entire  Decadal 
campaign  in  45  years.  Similarly,  doubling  the  budget  from  $300M  to  $600M  results  in  the 
larger-budget  campaign  capturing  as  much  benefit  in  less  than  ten  years  than  the  smaller 
budget  captures  in  45  years. 
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Cumulative  Percentage  of  Depreciated  Value  Accrued 


6  Conclusions  and  Future  Work 


Chapter  6  provides  conclusions  and  recommendations  for  Decadal  campaign 
architecture  development  using  the  CSTM.  Additionally,  areas  for  future  work  and  model 
improvement  are  identified. 

6.1  Conclusions 

This  thesis  has  shown  that  system  architecting  of  the  Decadal  campaign  can 
realistically  reproduce  the  decision  logic  of  the  Decadal  Survey,  while  accurately  capturing 
the  necessary  constraints  and  value  functions  in  an  automated  manner.  This  capability 
provides  decision  makers  a  key  tool  for  dealing  with  uncertainty  by  enabling  to  evaluate 
the  impacts  of  decisions  with  respect  to  the  entire  campaign. 

This  thesis  illustrated  a  technique  for  tracing  stakeholder  value  to  campaign 
architecture  decisions  through  the  use  of  science  traceability.  A  framework  for  campaign 
analysis  was  presented  and  applied  to  the  Decadal  campaign.  Relationships  between 
campaign  elements  were  enumerated  using  stakeholder  modeling,  the  Decadal  Survey,  and 
surveys  of  NASA  scientists  and  engineers.  This  model  for  tracing  value,  the  CSTM,  was  then 
validated  against  the  Decadal  Survey. 

The  CSTM  led  to  several  observations  about  the  Decadal  campaign.  First,  although 
each  of  the  17  proposed  missions  are  "research"  missions,  there  are  significantly 
differences  in  the  level  of  benefit  expected  from  each  mission.  While  the  Decadal  Survey 
does  not  explicitly  consider  the  value  of  each  mission,  it  may  be  desirable  to  apply  the  value 
traceability  as  a  constraint  in  scheduling.  Secondly,  there  is  a  disconnect  between  the 
objectives  of  the  Decadal  Survey  and  the  missions  proposed  to  accomplish  those  objectives. 

Additionally,  this  thesis  presented  a  refinement  for  a  technique  for  scheduling 
space-based  earth  observation  campaigns.  The  decision  logic  of  the  Decadal  Survey  was 
captured  through  the  development  of  constraints  and  value  functions,  which,  applied  by  an 
algorithm,  allow  the  systematic  design  and  evaluation  of  a  large  number  of  possible 
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solutions.  This  algorithm  was  validated  against  the  logic  and  sequence  proposed  in  the 
Decadal  Survey. 

The  CSTM  scheduling  algorithm  reflects  three  primary  criteria:  TRL,  cost,  and 
fairness.  TRL  is  used  to  ensure  individual  instrument  development  does  not  negatively 
affect  the  entire  campaign.  Cost  is  used  to  mitigate  the  risks  of  mission  cost  overruns. 
Fairness  is  used  to  ensure  that  different  sciences  communities  are  equally  satisfied  over  the 
course  of  the  campaign. 

Finally,  this  thesis  examined  the  impacts  of  key  changes  that  have  occurred  since  the 
publishing  of  the  Decadal  Survey  to  provide  insight  and  recommendations  for  the  earth 
observation  program.  Several  Scenarios  were  presented: 

•  The  campaign  sequence  proposed  by  the  Decadal  Survey  was  compared  to 
the  sequence  generated  using  the  latest  cost  and  TRL  assumptions  with  the 
CSTM  algorithm.  This  simulation  revealed  the  need  to  consider  benefit  in 
campaign  design. 

•  The  campaign  generated  with  the  latest  cost  and  TRL  assumptions  of  the  17 
Decadal  missions  was  compared  to  a  campaign  of  missions  in  which  the 
instrument  pairing  of  the  17  missions  were  broken  apart.  This  simulation 
revealed  that  there  are  significant  benefits  associated  with  flying  smaller 
missions. 

•  The  campaign  generated  with  the  updated  set  of  17  missions  was  compared 
to  a  corresponding  campaign  generated  from  the  repaired  instruments 
mission  set  to  analyze  the  impacts  of  campaign  decision  that  have  already 
been  made.  This  simulation  revealed  that  the  current  choice  of  missions  may 
not  be  optimally  suited  to  the  delivery  of  value. 

•  The  sensitivity  of  value  delivery  to  campaign  budget  was  analyzed.  This 
quantified  the  desirability  of  an  increased  budget  by  presenting  the  loss  to 
campaign  value  implicit  in  having  a  smaller  annual  budget. 
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6.2  Future  Work  with  Earth  Observation  Systems 

This  section  describes  the  areas  for  future  work  identified  in  previous  chapters. 
Preliminary  approaches  are  suggested. 


Revise  measurements  and  measurement  mappings:  The  instrument-measurement  mapping 
process  revealed  several  areas  where  the  common  set  of  measurements  could  be  expanded 
or  improved.  Several  areas  were  identified  where  measurements  can  be  combined,  such  as 
the  vegetation  measurements;  and  several  new  measurements  were  identified. 
Additionally,  the  GACM  instrument  mapping  needs  to  be  completed.  Revised  surveys  can 
be  complete  by  working  in  conjunction  with  NASA  Goddard. 


Investigate  the  contributions  of  International  space  programs:  The  CSTM  methodology 
allows  for  the  easy  inclusion  of  international  mission  through  the  measurement 
framework.  The  CEOS  database  can  be  utilized  to  identify  substitute  instruments  which 
capture  the  requisite  measurements.  This  information  can  be  utilized  to  inform  synergistic 
scheduling,  and  in  some  cases,  allow  for  the  demanifestation  of  a  particular  mission  to 
constraint  the  size  of  the  campaign 


Expand  the  solution  space  through  instrument-mission  architecting:  This  thesis  only 
analyzed  two  hand-crafted  mission  sets,  a  process  which  can  be  automated.  Given  a  known 
understanding  of  both  instrument  properties  and  requirements,  and  measurement 
synergies,  a  separate  mission  set  generator  can  be  developed  which  parametrically 
estimates  cost  parameters  for  new  missions.  This  could  be  combined  with  the  scheduling 
algorithm  to  identify  the  globally  optimum  missions  set  and  schedule. 


Explore  the  implementation  of  synergistic  measurement  qualities:  In  this  thesis  the 
quantification  of  measurement  synergies  was  identified  as  an  area  for  immediate  research. 
First  the  specific  relationships  amongst  measurements  must  be  captured.  Then  they  can  be 
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incorporated  into  the  CSTM  as  a  value  function  reflective  of  both  instrument-mission 
relationships  and  mission  scheduling.  A  survey  has  already  been  developed  to  quantify 
these  relationships,  although  it  has  not  been  completed. 


Expand  campaign  elements  to  include  ground  and  air  networks:  The  CSTM  framework  in 
this  thesis  is  only  applied  to  space-mission  campaign  elements.  It  is  desirable  to  expand 
the  framework  to  include  other  resources,  such  as  ground  and  air  observation  campaigns. 
This  will  require  the  development  of  separate  value  functions  and  constraints,  but  having 
multiple  campaign  elements  in  a  common  framework  will  allow  a  robust  generation  of 
campaign  architectures. 


Algorithm  experimentation:  The  results  in  Chapter  5  indicate  that  even  the  explicit  Decadal 
decision  rules  do  not  necessarily  reflect  the  tacit  Decadal  logic.  The  binning  of  missions 
into  three  tiers,  and  the  breakdown  of  those  tiers  when  considered  with  updated 
assumptions  indicates  a  more  sophisticated  decision  process.  Although  the  CSTM 
algorithm  was  validated  against  the  Decadal  schedule  with  the  Decadal  assumptions,  it  is 
desirable  to  revisit  different  algorithmic  considerations,  such  as  benefit  or  even  limited 
sequences  position  shifting. 


Computational  techniques:  The  results  generated  in  Chapter  5  were  all  products  of  a  manual 
implementation  of  the  CSTM  algorithm.  Several  techniques  are  being  explored  to  automate 
the  process  and  process  large  batches  of  possible  solutions.  This  includes  multi-objective 
optimization,  linear  programming,  and  the  use  of  genetic  algorithms  to  schedule  missions. 

It  is  desirable  to  be  able  to  both  enumerate  and  evaluate  large  numbers  of  feasible 
solutions,  so  that  the  global  maximum  can  be  identified. 
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7  Appendix  A:  NASA  Worksheet  Instructions 


Questionnaire 

There  are  three  parts  to  this  questionnaire.  The  first  section  is  designed  to 
understand  the  Decadal  Survey  Missions  as  originally  proposed.  The  second  section  is 
designed  to  understand  which  measurements  individual  instruments  are  taking.  The  third 
section  is  designed  to  understand  the  synergistic  effects  of  taking  certain  measurements 
concurrently. 

Section  1:  Decadal  Baseline 

Instructions: 

The  attached  spreadsheet  contains  a  matrix  of  the  17  Decadal  Survey  Missions  and  a  list  of 
proposed  measurements.  Using  a  scale  of  0-4  (see  table  below)  please  rate  the  usability  of 
data  produced  by  this  Mission  with  regards  to  a  specific  measurement.  You  are  answering 
the  question  "How  well  does  this  Mission  produce  measurements  of  this  type",  so  please 
consider  the  output  of  the  Mission  as  a  whole. 

The  usability  of  data  produced  is  a  combination  of  both  amount  and  quality.  If  a 
Mission  produces  a  combination  of  amount  and  quality  not  listed,  pick  the  lower  scoring 
option.  Presumably,  most  of  the  Decadal  Survey  Missions  are  optimized  to  produce  large 
amounts  of  high  quality  data  for  the  specific  measurements  they  were  designed  to  produce; 
however  it  is  possible  they  can  produce  secondary  measurements  in  a  sub-optimal  manner. 
You  will  notice  that  the  measurements  are  decomposed  into  three  layers  of  abstractions: 

Science  categories  (i.e.  1.  Atmosphere,  2.  Land,  3.  Ocean,  etc) 

General  measurements  (i.e.  1.1  Aerosol  Properties,  1.2  Atmospheric  temp  fields  etc) 

Specific  measurements  (i.e.  1.1.1  Aerosol  height/optical  depth,  etc) 

Please  rate  each  mission  to  the  lowest  level  of  abstraction  that  you  are  able.  If  you 
feel  a  measurement  is  missing  from  the  list,  please  add  it  to  the  bottom  of  the  matrix  and  fill 
in  accordingly  for  all  17  Missions,  as  well  as  annotating  where  it  should  fall  into  this 
hierarchy. 
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Usability  of  Data  Produced 

Score 

This  Mission  produces  no  data  for  this  measurement 

0 

This  Mission  produces  low  quality  data  for  this  measurement  OR 
this  Mission  produces  a  small  amount  of  data  for  this  measurement 

1 

This  Mission  produces  moderate  quality  data  for  this  measurement  OR 
this  Mission  produces  a  moderate  amount  of  data  for  this  measurement 

2 

This  Mission  produces  high  quality  data  for  this  measurement  OR 
this  Mission  produces  a  large  amount  of  data  for  this  measurement 

3 

This  Mission  produces  the  highest  possible  quality  data  for  this 
measurement  AND 

this  Mission  produces  a  large  amount  of  data  for  this  measurement 

4 
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Part  2:  Instrument  Baseline 


Instructions: 

The  second  tab  on  the  spreadsheet  contains  a  matrix  of  the  individual  instruments 
proposed  in  the  Decadal  Survey  Missions  and  a  list  of  measurements.  Using  a  scale  of  0-4 
(see  table  below)  please  rate  the  usability  of  data  produced  by  this  Instrument  with 
regards  to  a  specific  measurement.  You  are  scoring  Instruments  as  isolated  things: 
consider  only  the  measurements  produced  by  this  specific  Instrument.  You  are  answering 
the  question  "How  well  does  this  Instrument  produce  measurements  of  this  type".  The 
usability  of  data  produced  is  a  combination  of  both  amount  and  quality.  If  an  Instrument 
produces  a  combination  of  amount  and  quality  not  listed,  pick  the  lower  scoring  option. 

Unlike  in  Section  1,  it  may  be  unlikely  that  these  Instruments  are  optimized  for 
certain  measurements,  and  instead  rely  upon  synergistic  effects  (which  will  be  captured  in 
section  3)  to  create  an  optimal  measurements.  Please  do  your  best  to  capture  the  Usability 
of  each  instrument  in  isolation.  If  you  added  any  measurements  in  Section  1,  please  add 
them  to  this  list  as  well. 

Additionally,  for  each  instrument  please  record  the  expected  per  unit  cost  of  the 
instrument.  Space  has  been  provided  to  do  this. 


Usability  of  Data  Produced 

core 

This  Instrument  produces  no  data  for  this  measurement 

This  Instrument  produces  low  quality  data  for  this  measurement  OR 
this  Instrument  produces  a  small  amount  of  data  for  this  measurement 

This  Instrument  produces  moderate  quality  data  for  this  measurement  OR 
this  Instrument  produces  a  moderate  amount  of  data  for  this  measurement 

This  Instrument  produces  high  quality  data  for  this  measurement  OR 
this  Instrument  produces  a  large  amount  of  data  for  this  measurement 

This  Instrument  produces  the  highest  possible  quality  data  for  this 
measurement  AND  this  Instrument  produces  a  large  amount  of  data  for  this 
measurement 
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Part  3:  Measurement  Synergies 

Instructions: 

The  following  pages  contain  matrices  of  measurements  correlated  against  other 
measurements.  This  section  is  intended  to  capture  the  synergistic  science  benefits  to 
having  concurrent  measurements.  Using  a  scale  of  0-4  (see  table  below)  please  rate  the 
increase  in  usability  of  data  of  one  measurement  when  complimentary  measurements  are 
made.  You  are  answering  the  question  "How  does  this  measurement  benefit  from  the 
presence  of  another  measurement". 

If  you  added  any  measurements  in  Sections  1  and  2,  please  add  them  to  this  matrix 
as  well.  Looking  at  the  matrices,  you  are  evaluating  the  affects  of  the  columns  upon  the 
rows,  that  is,  "assuming  you  have  the  measurement  in  a  particular  horizontal  row,  and 
someone  were  to  give  you  the  data  from  the  measurement  is  the  vertical  column,  how 
would  it  change  the  usability  of  your  measurement".  Because  of  the  different  layer  of 
abstraction  being  used  in  this  survey,  once  again  please  fill  in  the  lowest  level  possible. 
We'd  ideally  like  every  single  white  colored  cell  to  be  filled  in,  even  if  they  are  mostly  zeros. 

It  is  expected  that  most  Measurements  will  not  be  affected  by  Complimentary 
Measurements  (i.e.  3.3.1  Ocean  salinity  is  not  affected  by  4.1.4  Ice  sheet  velocity)  and  will 
score  zeros,  however,  some  Measurements  may  be  entirely  derived  from  combinations  of 
Complimentary  Measurement  (and  would  hence  score  fours).  Please  choose  the  score  that 
best  captures  the  positive  complimentary  effects. 


Usability  of  Measurement 

core 

This  Measurement  is  not  affected  by  this  Complimentary  Measurement 

This  Measurement  is  slightly  more  useable  with  the  addition  of  this 
Complimentary  Measurement 

This  Measurement  is  moderately  more  useable  with  the  addition  of  this 
Complimentary  Measurement 

This  Measurement  is  significantly  more  useable  with  the  addition  of  this 
Complimentary  Measurement 

This  Measurement  completely  requires  the  addition  of  this  Complimentary 
Measurement 
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8  Appendix  B:  Reference  Sequence 


TABLE  2.3  Contribution  of  Recommended  Missions  to  the  Priority  Science  Mission/Observation  Types 
Identified  by  the  Individual  Study  Panels  as  Discussed  in  Part  III 

Recommended  Mission 

Mission/Observation  Type  Recommended  by  Individual  Panel 

Panel 

CLARREO 

Radiance  calibration 

Climate 

Ozone  processes 

Health 

GPSRO 

Radiance  calibration 

Climate 

Ozone  processes 

Health 

Cold  seasons 

Water 

Radio  occupation 

Weather 

SMAP 

Heat  stress  and  drought 

Health 

Algal  blooms  and  wraterborne  infectious  disease 

Health 

Vector-borne  and  zoonotic  disease 

Health 

Soil  moisture  and  freeze-thaw  state 

Water 

Surface  water  and  ocean  topography 

Water 

ICESat-ll 

Clouds,  aerosols,  ice,  and  carbon 

Climate 

Ecosystem  structure  and  biomass 

Ecosystem 

Sea  ice  thickness,  glacier  surface  elevation,  glacier  velocity 

Water 

DESDynl 

Ice  dynamics 

Climate 

Ecosystem  structure  and  biomass 

Ecosystem 

Heat  stress  and  drought 

Health 

Vector-borne  and  zoonotic  disease 

Health 

Surface  deformation 

Solid  Earth 

Sea  ice  thickness,  glacier  surface  elevation,  glacier  velocity 

Water 

XOVWM 

Ocean  circulation,  heat  storage,  and  climate  forcing 

Climate 

HyspIRI 

Ecosystem  function 

Ecosystem 

Heat  stress  and  drought 

Health 

Vector-borne  and  zoonotic  disease 

Health 

Surface  composition  and  thermal  properties 

Solid  Earth 

ASCENDS 

Carbon  budget 

Ecosystem 

Ozone  processes 

Health 

SWOT 

Ocean  circulation,  heat  storage,  and  climate  forcing 

Climate 

Algal  blooms  and  waterborne  infectious  disease 

Health 

Vector-borne  and  zoonotic  disease 

Health 

Surface  water  and  ocean  topography 

Water 

GEO-CAPE 

Global  ecosystem  dynamics 

Ecosystem 

Ozone  processes 

Health 

Heat  stress  and  drought 

Health 

Acute  toxic  pollution  releases 

Health 

Air  pollution 

Health 

Algal  blooms  and  waterborne  infectious  disease 

Weather 

Inland  and  coastal  water  quality 

Health 

Tropospheric  aerosol  characterization 

Water 

Tropospheric  ozone 

Weather 
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TABLE  2.3  Continued 


Recommended  Mission 

Mission/Observation  Type  Recommended  by  Individual  Panel 

Panel 

ACE 

Clouds,  aerosols,  ice,  and  carbon 

Climate 

Ice  dynamics 

Climate 

Global  ocean  productivity 

Ecosystem 

Ozone  processes 

Health 

Acute  toxic  pollution  releases 

Health 

Air  pollution 

Health 

Algal  blooms  and  Viraterborne  infectious  disease 

Health 

Aerosol-cloud  discovery 

Weather 

Tropospheric  aerosol  characterization 

Weather 

Tropospheric  ozone 

Weather 

LIST 

Heat  stress  and  drought 

Health 

Vector-borne  and  zoonotic  disease 

Health 

High-resolution  topography 

Solid  Earth 

PATH 

Heat  stress  and  drought 

Health 

Algal  blooms  and  waterborne  infectious  disease 

Health 

Vector-borne  and  zoonotic  disease 

Health 

Cold  seasons 

Water 

All-weather  temperature  and  humidity  profiles 

Weather 

GRACE-II 

Ocean  circulation,  heat  storage,  and  climate  forcing 

Climate 

Groundwater  storage,  ice  sheet  mass  balance,  ocean  mass 

Water 

SCLP 

Cold  seasons 

Water 

3D- Winds 

Water  vapor  transport 

Water 

Tropospheric  winds 

Weather 

GACM 

Global  ecosystem  dynamics 

Ecosystem 

Ozone  processes 

Health 

Acute  toxic  pollution  releases 

Health 

Air  pollution 

Health 

Cold  seasons 

Water 

Tropospheric  aerosol  characterization 

Weather 

Tropospheric  ozone 

Weather 

Figure  50.  Reference  Sequence  (National  Research  Council,  2007) 
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9  Appendix  C:  Measurement  Weights 


1)  Atmosphere 

17.8451909 

1.1.  Aerosol  properties 

4.17064599 

1.1.1  aerosol  height/optical  depth 

1.18693182 

1.1.2  aerosol  composition,  physical  and  chemical 
properties 

1.01646373 

1.1.3  aerosol  scattering  properties 

0.47539336 

1.1.4  aerosol  extinction  profiles 

0.47539336 

1.1.5  aerosol  size  and  size  distribution 

1.01646373 

1.2.  Atmospheric  temperature  fields 

0.96094872 

1.2.1  Atmospheric  temperature  fields 

0.96094872 

1.3.  Water  vapor 

2.07790028 

1.3.1  Water  vapor  profiles 

1.54489052 

1.3.2  Water  vapor  transport 

0.53300977 

1.4.  Atmospheric  \winds 

1.18114774 

1.4.1  atmospheric  wind  speed 

0.59057387 

1.4.2  atmospheric  wind  direction 

0.59057387 

1.5.  Cloud  type,  amount,  and  top  temp. 

0.68854312 

1.5.1  cloud  top  temperature 

0.2125 

1.5.2  Cloud  type 

0.20923951 

1.5.3  Cloud  amount/distribution 

0.26680361 

1.6.  Cloud  particle  properties  and  profile 

0.71086668 

1.6.1  cloud  height/optical  thickness 

0.20923951 

1.6.2  cloud  particle  size  distribution 

0.29238766 

1.6.3  ice/water  transition  in  clouds 

0.20923951 

1.7.  Liquid  water  and  precipitation  rate 

0.84506002 

1.7.1  Precipitation  rate 

0.72700641 

1.7.2  droplet  size 

0.11805361 

1.8.  Ozone 

0.87325499 

1.8.1  stratospheric  ozone 

0.08314815 

1.8.2  tropospheric  ozone 

0.62376068 

1.8.3  ozone  precursors 

0.16634615 

1.9.  Radiation  budget 

2.37740456 

1.9.1  total  solar  irradiance 

0.833 

1.9.2  spectrum  of  earth  IR  radiance 

0.97371225 

1.9.3  GPS  radio  occultation 

0.57069231 

1.10.  Trace  gases  (excluding  ozone) 

3.9594188 

1.10.1  short-lived  reactive  species  (OH,  H02,  N02, 
CIO,  BrO,  10,  H0N02,  HCI,  CH20) 

0.08314815 

1.10.2  isotope  observations  (HDO,  H2180,  H20) 

0.08314815 

1.10.3  tropospheric  column  502,  N02, 
formaldehyde 

0.62376068 

2)  Land 

11.47027 

2.1.  Albedo  and  reflectance 

0 

2.1.1  Albedo  and  reflectance 

0 

2.2.  Land  topography 

0.889365 

2.2.1  surface  deformation 

0.307857 

2.2.2  Hi-res  topography 

0.581508 

2.3.  Soil  moisture 

2.139607 

2.3.1  Freeze/thaw  state 

0.901429 

2.3.2  soil  moisture 

1.238179 

2.4.  Vegetation 

3.55655 

2.4.1  vegetation  type 

0.2125 

2.4.2  vegetation  state 

1.58625 

2.4.3  vegetation  height 

0.8789 

2.4.4  canopy  density 

0.8789 

2.5.  Surface  temperature  (land) 

0.667673 

2.5.1  Surface  temperature  (land) 

0.667673 

2.6.  Multi-purpose  imagery  (land) 

3.15516 

2.6.1  land  use 

0.060095 

2.6.2  landcover  status 

0.792862 

2.6.3  disaster  monitoring 

1.421156 

2.6.4  hydrocarbon  reservoir  monitoring 

0.564405 

2.6.5  surface  composition 

0.256548 

2.6.6  inland  water  quality 

0.060095 

2.7  Surface  water  distribution 

1.061918 

2.7.1  river  and  lake  elevation 

0.500794 

2.7.2  flood  monitoring 

0.273651 

2.7.3  groundwater  storage 

0.287474 

3)  Ocean 

6.65481 

3.1.  Ocean  colour/biology 

1.294709 

3.1.1  Ocean  color 

1.294709 

3.2.  Ocean  topography/currents 

3.024277 

3.2.1  surface  circulation 

0.907817 

3.2.2  seafloor  topography 

0.407024 

3.2.3  coastal  upwelling 

0.444267 

3.2.4  thermal  plumes 

0.187083 

3.2.5  river  plumes/sediment  fluxes 

0.691445 

3.2.6  Ocean  mass  distribution 

0.38664 

3.3.  Ocean  salinity 

0 
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1.10.4  Benchmark  tracer  data  (C02,  CO,  HD0/H20, 
NOy,  N20,  CH4,  halogen  source  molecules) 

0.08314815 

1.10.5  visible  atmospheric  plumes 

0.0575641 

1.10.6  pollutant  particle  size 

0.0575641 

1.10.7  pollutant  gross  vertical  structure 

0.0575641 

1.10.8  vertically  resolved  CO  (Daytime  only) 

1.04026068 

1.10.9  CO  vertically  resolved  CO  (Day/Night) 

1.04026068 

1.10.11  C02  concentrations 

0.4165 

1.10.12  02  concentrations 

0.4165 

4)  Snow  and  Ice 

4.29579696 

4.1.  Ice  sheet  topography 

2.09974537 

4.1.1  ice  sheet  volume 

0.371875 

4.1.2  Glacier  surface  elevation 

0.49707341 

4.1.3  glacier  mass  balance 

0.65934854 

4.1.4  Ice  sheet  velocity 

0.19957341 

4.1.5  Ice  Sheet  topography 

0.371875 

4.2.  Snow  cover,  edge  and  depth 

1.49940476 

4.2.1  snow-water  equivalence 

0.37485119 

4.2.2  snow  depth 

0.37485119 

4.2.3  snow  wetness 

0.37485119 

4.2.4  snow  cover 

0.37485119 

4.3.  Sea  ice  cover,  edge  and  thickness 

0.69664683 

4.3.1  Sea  ice  thickness 

0.49707341 

4.3.2  Sea  ice  cover 

0.19957341 

3.3.1  Ocean  salinity 

0 

3.4.  Ocean  surface  winds 

0.863718 

3.4.1  Ocean  surface  wind  speed 

0.431859 

3.4.2  Ocean  surface  wind  direction 

0.431859 

3.5.  Surface  temperature  (ocean) 

0.831442 

3.5.1  Surface  temperature  (ocean) 

0.831442 

3.6.  Ocean  wave  height  and  spectrum 

0 

3.6.1  Ocean  wave  height  and  spectrum 

0 

3.7.  Multi-purpose  imagery  (ocean) 

0.640664 

3.7.1  visible  hydrospheric  pollution 
plumes 

0.057564 

3.7.2  coral  reef  health/extent 

0.5831 

5)  Gravity  and  Magnetic  fields 

0.153929 

5.1.  Gravity,  magnetic  and  geodynamic 

0.153929 

5.1.1  gravity  field  variations 

0.153929 

5.1.2  magnetic  field  variations 

0 
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